#ceph IRC Log

Index

IRC Log for 2014-09-06

Timestamps are in GMT/BST.

[0:10] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[0:10] * dupont-y (~dupont-y@2a01:e34:ec92:8070:3400:81b0:ef39:d655) Quit (Quit: Ex-Chat)
[0:14] * Nacer (~Nacer@2001:41d0:fe82:7200:583a:d0d2:1fd6:22db) Quit (Remote host closed the connection)
[0:15] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[0:23] * rendar (~I@host174-179-dynamic.7-87-r.retail.telecomitalia.it) Quit ()
[0:27] <JoeJulian> I notice there's no fedora 20 support in http://ceph.com/rpm-emperor which I assume is because the packages are part of the fedora base. What's not part of base, though, is ceph-deploy - something that also doesn't function with fedora 20 since it tries to replace existing packages with the non-existent emperor ones. I don't see any quick-starts for Fedora 20 anywhere. Is fedora 20 unsupported?
[0:30] * reed (~reed@75-101-54-131.dsl.static.sonic.net) Quit (Remote host closed the connection)
[0:35] * JayJ__ (~jayj@157.130.21.226) Quit (Quit: Computer has gone to sleep.)
[0:37] * marrusl (~mark@2604:2000:60e3:8900:409c:a141:606f:ff70) Quit (Remote host closed the connection)
[0:38] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[0:45] * fsimonce (~simon@host135-17-dynamic.8-79-r.retail.telecomitalia.it) Quit (Quit: Coyote finally caught me)
[0:47] * dmsimard is now known as dmsimard_away
[0:47] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[0:49] * marrusl (~mark@2604:2000:60e3:8900:2dc1:c0bb:8e91:5b2d) has joined #ceph
[0:50] * angdraug (~angdraug@12.164.168.117) Quit (Remote host closed the connection)
[0:51] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[0:52] * alram_ (~alram@38.122.20.226) Quit (Quit: leaving)
[0:52] * PerlStalker (~PerlStalk@2620:d3:8000:192::70) Quit (Quit: ...)
[0:58] <dmick> carmstrong: i would imagine at least logs; don't know what else
[0:58] * sz0 (~sz0@94.55.197.185) has joined #ceph
[1:07] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[1:09] * steki (~steki@77.243.20.78) has joined #ceph
[1:09] * reed (~reed@75-101-54-131.dsl.static.sonic.net) has joined #ceph
[1:12] * BManojlovic (~steki@77.243.20.79) Quit (Ping timeout: 480 seconds)
[1:13] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Quit: Leaving.)
[1:17] <carmstrong> does anyone know if ceph.conf is ever consulted to determine how to connect to monitors, or if osds always use the monmap?
[1:18] <carmstrong> I'm thinking that the conf should define all monitor hosts, so that clients can connect to any of them should one of them go down
[1:19] <carmstrong> dmick: thanks. I'll have to play around with it and see if blowing away the data directory on a gateway has any repercussions
[1:19] <carmstrong> I"m writing logs to /dev/stdout on the gateway (it runs in a container)
[1:21] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[1:27] * ccheng_ (~ccheng@128.211.165.1) has joined #ceph
[1:29] <carmstrong> I guess by "client" in this context, I'm mostly referring to radosgw. we only have osds, mons, and a radosgw in the cluster
[1:29] * steki (~steki@77.243.20.78) Quit (Ping timeout: 483 seconds)
[1:34] * ccheng (~ccheng@csdhcp-120-248.cs.purdue.edu) Quit (Ping timeout: 480 seconds)
[1:35] <dmick> carmstrong: well clients have to start with ceph.conf of course
[1:35] * ccheng_ (~ccheng@128.211.165.1) Quit (Ping timeout: 480 seconds)
[1:36] <dmick> and yeah, that's a good reason to list them all
[1:36] <dmick> and the other client is "ceph" itself
[1:36] <dmick> (or rados, or radosgw-admin)
[1:38] <carmstrong> ok. so if I have the luxury, I should not use initial monitors in my ceph.conf, and instead write them all out there from the get-go
[1:38] <carmstrong> but what if they're not all up yet?
[1:38] <carmstrong> I suppose I could a) write the intial config with mon.1, start mon.1; b) update ceph.conf to include mons 2 and 3; c) start mons 2 and 3
[1:39] * dneary (~dneary@96.237.180.105) has joined #ceph
[1:39] <dmick> it's complicated and I'm not 100% solid on this
[1:39] <dmick> but there are at least two config items
[1:41] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[1:47] * alram (~alram@cpe-172-250-2-46.socal.res.rr.com) has joined #ceph
[1:48] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[1:49] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[1:49] <carmstrong> hmm
[1:51] * scuttlemonkey is now known as scuttle|afk
[1:51] * ircolle (~Adium@c-67-172-132-222.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[1:57] <dmick> sorry to disappear on you. what I mean are
[1:57] <dmick> mon_initial_members
[1:57] <dmick> and mon_host
[1:58] * dneary (~dneary@96.237.180.105) Quit (Read error: Operation timed out)
[1:58] <dmick> (and of course the monmap state in the system)
[1:58] * dmsimard_away is now known as dmsimard
[1:58] <dmick> and I can never remember exactly how they interact
[1:59] <joshd> mon_initial_members is just for telling the monitors who's supposed to be available before they form their first quorum, so you don't end up with 3 1-mon clusters
[1:59] * bandrus (~Adium@216.57.72.205) has left #ceph
[1:59] <joshd> mon_host is the usual setting consulted for the mon addresses by clients
[2:00] <tab_> in case I have 3 disks within PG. than one disk within this PG dies. Is CEPH continue to write to that PG or it waits for some other disk to be assigned to PG?
[2:01] <carmstrong> joshd: ok. so I'll definitely want to add mon_host to the config file once all the mons are up
[2:01] <joshd> carmstrong: yeah
[2:01] * marrusl (~mark@2604:2000:60e3:8900:2dc1:c0bb:8e91:5b2d) Quit (Quit: sync && halt)
[2:01] * marrusl (~mark@2604:2000:60e3:8900:2dc1:c0bb:8e91:5b2d) has joined #ceph
[2:02] <dmick> note that you *can* specify on the cmdline too to dig out of jams
[2:02] <dmick> reasonably involved discussion at http://ceph.com/docs/firefly/dev/mon-bootstrap/
[2:02] <joshd> tab_: by default yes, it's controlled by http://ceph.com/docs/master/rados/operations/pools/?highlight=min_size#set-pool-values
[2:07] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[2:07] * TiCPU (~jeromepou@190-130.cgocable.ca) Quit (Quit: Ex-Chat)
[2:07] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[2:07] <tab_> joshd: so in case I have 3 disks within PG, and set min_size=3 as num_replicase=3, then I won't be able to write to that PG? what about option to write to other PG temporary and than files would be copied to failed PG , when disk is changed for new one?
[2:09] * rmoe (~quassel@12.164.168.117) Quit (Read error: Operation timed out)
[2:09] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[2:11] * johntwilkins (~john@c-50-131-97-162.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[2:13] <dmick> I feel like it's important to note that "disks aren't within pgs", but rather "pgs are a grouping of cluster objects that are replicated across many disks"
[2:13] <dmick> if your current live set is less than the min_size, you won't be able to read or write objects in that PG until the cluster fixes that
[2:14] <dmick> the usual way is for another OSD to be assigned to the set of OSDs responsible for replicating that pg
[2:14] <dmick> so the temporary thing is "temporarily replicating this PG to a,b,c' rather than the original a,b,c"
[2:15] <dmick> until the temporary is allocated and gets all the data, the PG is still inactive
[2:15] <dmick> (at least I think that's right)
[2:17] * sjustwork (~sam@2607:f298:a:607:f88e:c8d2:ee4c:2534) Quit (Quit: Leaving.)
[2:17] * jobewan (~jobewan@c-75-65-191-17.hsd1.la.comcast.net) has joined #ceph
[2:20] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[2:20] <tab_> dmick: thx. another OSD to be assigned is than manual process, right?
[2:21] * yanzheng (~zhyan@171.221.139.239) Quit ()
[2:21] <joshd> no that's automatic
[2:22] <tab_> so some unformated disks within node, becomes available and in use for that certain PG?
[2:23] <tab_> let's say i leave some disks within node unused from the first installation of ceph....
[2:25] * rmoe (~quassel@173-228-89-134.dsl.static.sonic.net) has joined #ceph
[2:26] <joshd> no, there are many pgs on a disk
[2:27] <joshd> this explains a bit about pgs and failure handling: http://ceph.com/docs/master/architecture/#mapping-pgs-to-osds
[2:28] <joshd> there needs to be free space on an osd to get new copies of data, of course, but that doesn't mean empty disks
[2:28] <tab_> ok. thx
[2:31] * tab_ (~oftc-webi@89-212-99-37.dynamic.t-2.net) Quit (Remote host closed the connection)
[2:34] * alram (~alram@cpe-172-250-2-46.socal.res.rr.com) Quit (Quit: leaving)
[2:36] * xarses (~andreww@12.164.168.117) Quit (Ping timeout: 480 seconds)
[2:42] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[2:44] * dmsimard is now known as dmsimard_away
[2:44] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[2:45] * monsterz_ (~monsterzz@94.19.146.224) has joined #ceph
[2:50] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[2:53] * monsterz_ (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[2:55] * jobewan (~jobewan@c-75-65-191-17.hsd1.la.comcast.net) Quit (Quit: Leaving)
[3:03] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Quit: Leaving.)
[3:04] <carmstrong> joshd: it looks like all of my rbd issues are fixed on the new kernel CoreOS released today
[3:05] <carmstrong> we're now on 3.16.1
[3:07] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[3:07] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit ()
[3:12] * xarses (~andreww@c-76-126-112-92.hsd1.ca.comcast.net) has joined #ceph
[3:14] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[3:17] * adamcrume_ (~quassel@2601:9:6680:47:1954:4d59:7b88:2d33) Quit (Remote host closed the connection)
[3:20] * Sysadmin88 (~IceChat77@2.124.167.78) Quit (Ping timeout: 480 seconds)
[3:26] * danieagle (~Daniel@179.184.165.184.static.gvt.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[3:39] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[3:39] * sz0 (~sz0@94.55.197.185) Quit ()
[3:42] * dmsimard_away is now known as dmsimard
[3:46] * cmendes0101 (~cmendes01@pool-98-112-76-22.lsanca.fios.verizon.net) has joined #ceph
[3:46] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[3:48] * dmsimard is now known as dmsimard_away
[3:51] <cmendes0101> On a setup I have for one machine 1 ssd and 3 sata drives. Would I install the os on the ssd then make a partition on there to also use that for journaling?
[3:54] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[3:58] * jamin (~jamin@65-100-221-49.dia.static.qwest.net) has joined #ceph
[4:02] <carmstrong> joshd: okay so using `rbd map` in teh container still results in the invalid argument error, but if I echo into the system bus to create the device on the host OS, the device is accessible within the container
[4:02] <carmstrong> and I can mount it, make a filesystem, etc.
[4:03] <carmstrong> this is with auth disabled
[4:03] <carmstrong> not sure if that's the best way to use it, but it works
[4:03] <kraken> ???_???
[4:04] <jamin> I've had ceph running with two storage nodes and one monitor now for a while. I recently added a third storage node and changed the replica size from 2 to 3. However, I noticed that OSDs appear to be going down and then popping back up. Checking the host I see lots of messages like the following: ceph-osd (ceph/11) main process (27218) killed by ABRT signal
[4:04] <carmstrong> jamin: no idea, but http://tracker.ceph.com/issues/6386 seems related
[4:05] <jamin> carmstrong: yea, found that in googling, but I don't have any hung tasks
[4:05] <jamin> the ceph-osd process just runs for a while, gets the ABRT, terminates, and upstart restarts it
[4:06] <jamin> each ceph-osd restart (or terminate, not sure which) is dumping a lot of data into the log file causing it to grow quite quickly too
[4:13] * blahnana (~bman@104-97-248-162-static.reverse.queryfoundry.net) Quit (Remote host closed the connection)
[4:13] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[4:28] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[4:34] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[4:34] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit ()
[4:36] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[4:38] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Remote host closed the connection)
[4:38] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[4:44] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) has joined #ceph
[4:46] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[4:47] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[4:53] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[4:53] <rkdemon> Hi: I asked earlier but netchat logged me out so if someone responded I didn't get it..
[4:54] <rkdemon> Is there a good pointer to going about installing calamari on ubuntu trust for my 3 node ceph cluster ?
[4:55] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[4:57] <rkdemon> anyone?
[4:59] <jamin> rkdemon: I haven't touched calamari yet, no idea
[4:59] <rkdemon> i got the ceph clusterup y'day with lurbs' help but the goal was stretched to have calamri else its no good for m boss :-(
[5:02] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[5:06] * yanzheng (~zhyan@171.221.139.239) Quit ()
[5:10] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[5:12] <isodude> I was tinkering with the idea of running calamari on a vm
[5:12] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Remote host closed the connection)
[5:12] <isodude> but I'm scared of it since we also run salt on the ceph nodes
[5:13] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[5:14] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Remote host closed the connection)
[5:15] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[5:15] * jdillaman (~jdillaman@pool-108-18-232-208.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[5:16] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) Quit (Ping timeout: 480 seconds)
[5:17] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[5:22] * nitti (~nitti@199.59.119.21) has joined #ceph
[5:22] * codice_ (~toodles@97-94-175-73.static.mtpk.ca.charter.com) has joined #ceph
[5:23] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[5:23] * LeaChim (~LeaChim@host81-159-253-189.range81-159.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[5:23] * Vacuum (~vovo@88.130.212.187) has joined #ceph
[5:24] * codice (~toodles@97-94-175-73.static.mtpk.ca.charter.com) Quit (Ping timeout: 480 seconds)
[5:25] * rotbeard (~redbeard@2a02:908:df19:5000:76f0:6dff:fe3b:994d) has joined #ceph
[5:26] <rkdemon> isodude: i am planning on just that
[5:27] <rkdemon> I am trying to install calamari on my VM local to me on my laptop and see if I can access my servers in the Data center
[5:27] <rkdemon> There are 3 servers that are setup as ceph nodes
[5:29] * vbellur (~vijay@122.167.123.123) has joined #ceph
[5:30] * Pedras (~Adium@50.185.218.255) has joined #ceph
[5:30] * Vacuum_ (~vovo@i59F7AFE5.versanet.de) Quit (Ping timeout: 480 seconds)
[5:30] * jdillaman (~jdillaman@pool-108-18-232-208.washdc.fios.verizon.net) has joined #ceph
[5:35] * t0rn (~ssullivan@c-68-62-1-186.hsd1.mi.comcast.net) has joined #ceph
[5:35] * t0rn (~ssullivan@c-68-62-1-186.hsd1.mi.comcast.net) has left #ceph
[5:47] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[5:47] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[5:54] * nitti (~nitti@199.59.119.21) Quit (Remote host closed the connection)
[5:56] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[5:57] * nitti (~nitti@199.59.119.21) has joined #ceph
[6:05] * jdillaman (~jdillaman@pool-108-18-232-208.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[6:05] * nitti (~nitti@199.59.119.21) Quit (Ping timeout: 480 seconds)
[6:19] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[6:28] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[6:29] * yanzheng (~zhyan@171.221.139.239) Quit ()
[6:34] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[6:37] * jamin (~jamin@65-100-221-49.dia.static.qwest.net) has left #ceph
[6:44] * reed (~reed@75-101-54-131.dsl.static.sonic.net) Quit (Ping timeout: 480 seconds)
[6:48] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[6:56] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[7:04] * bipinkunal (~bkunal@113.193.137.1) has joined #ceph
[7:04] * bkunal (~bkunal@1.23.195.13) Quit (Ping timeout: 480 seconds)
[7:05] * MACscr (~Adium@c-98-214-170-53.hsd1.il.comcast.net) has left #ceph
[7:09] * cmendes0101 (~cmendes01@pool-98-112-76-22.lsanca.fios.verizon.net) Quit (Remote host closed the connection)
[7:32] * michalefty (~micha@ip25045ed2.dynamic.kabel-deutschland.de) has joined #ceph
[7:41] * bipinkunal (~bkunal@113.193.137.1) Quit (Ping timeout: 480 seconds)
[7:44] * michalefty (~micha@ip25045ed2.dynamic.kabel-deutschland.de) Quit (Quit: Leaving.)
[7:45] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[7:49] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[7:55] * zerick (~eocrospom@190.118.30.195) Quit (Ping timeout: 480 seconds)
[7:55] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[7:57] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[8:00] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[8:04] * KevinPerks (~Adium@2606:a000:80a1:1b00:456b:bf96:6ed7:2eb1) Quit (Quit: Leaving.)
[8:09] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) has joined #ceph
[8:33] * lupu (~lupu@86.107.101.214) Quit (Quit: Leaving.)
[8:44] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[8:45] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[8:50] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[8:52] * vbellur (~vijay@122.167.123.123) Quit (Ping timeout: 480 seconds)
[8:58] * monsterzz (~monsterzz@94.19.146.224) Quit (Ping timeout: 480 seconds)
[8:59] * fghaas (~florian@85-127-80-104.dynamic.xdsl-line.inode.at) has joined #ceph
[9:01] * jtaguinerd (~jtaguiner@112.205.24.200) has joined #ceph
[9:06] * jtaguinerd (~jtaguiner@112.205.24.200) Quit ()
[9:10] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Remote host closed the connection)
[9:11] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[9:18] * steveeJ (~junky@HSI-KBW-085-216-022-246.hsi.kabelbw.de) Quit (Remote host closed the connection)
[9:19] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[9:30] * zerick (~eocrospom@190.118.30.195) has joined #ceph
[9:32] * monsterzz (~monsterzz@94.19.146.224) has joined #ceph
[9:36] * monsterzz (~monsterzz@94.19.146.224) Quit (Read error: Connection reset by peer)
[9:40] * Sysadmin88 (~IceChat77@94.1.36.141) has joined #ceph
[9:40] * chowmeined (~chow@c-24-19-66-251.hsd1.wa.comcast.net) Quit (Quit: Leaving)
[9:59] * yguang11 (~yguang11@182.18.27.82) has joined #ceph
[10:01] * yguang11 (~yguang11@182.18.27.82) Quit (Remote host closed the connection)
[10:02] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) has joined #ceph
[10:03] * fghaas (~florian@85-127-80-104.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[10:11] * madkiss (~madkiss@46.115.162.71) has joined #ceph
[10:15] * rendar (~I@host215-182-dynamic.19-79-r.retail.telecomitalia.it) has joined #ceph
[10:18] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) Quit (Remote host closed the connection)
[10:18] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) has joined #ceph
[10:28] * linjan (~linjan@176.195.181.192) has joined #ceph
[10:33] * lalatenduM (~lalatendu@122.172.214.130) has joined #ceph
[10:35] * zerick (~eocrospom@190.118.30.195) Quit (Ping timeout: 480 seconds)
[10:36] * mongo_ (~gdahlman@voyage.voipnw.net) has joined #ceph
[10:36] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[10:38] * mongo (~gdahlman@voyage.voipnw.net) Quit (Ping timeout: 480 seconds)
[10:41] * yguang11_ (~yguang11@vpn-nat.corp.tw1.yahoo.com) has joined #ceph
[10:41] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) Quit (Read error: Connection reset by peer)
[10:44] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[10:46] * madkiss (~madkiss@46.115.162.71) Quit (Quit: Leaving.)
[10:50] * fghaas (~florian@85-127-80-104.dynamic.xdsl-line.inode.at) has joined #ceph
[10:54] * fghaas (~florian@85-127-80-104.dynamic.xdsl-line.inode.at) Quit ()
[10:57] * yguang11_ (~yguang11@vpn-nat.corp.tw1.yahoo.com) Quit (Ping timeout: 480 seconds)
[11:03] * Pedras (~Adium@50.185.218.255) Quit (Quit: Leaving.)
[11:19] * dignus (~jkooijman@t-x.dignus.nl) Quit (Ping timeout: 480 seconds)
[11:23] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) Quit (Read error: Connection reset by peer)
[11:31] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) has joined #ceph
[11:38] * steveeJ (~junky@HSI-KBW-085-216-022-246.hsi.kabelbw.de) has joined #ceph
[11:48] * zack_dolby (~textual@p843a3d.tokynt01.ap.so-net.ne.jp) Quit (Ping timeout: 480 seconds)
[11:54] * zack_dolby (~textual@p843a3d.tokynt01.ap.so-net.ne.jp) has joined #ceph
[12:07] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[12:16] * eoin_ (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) has joined #ceph
[12:16] * eoin_ (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) Quit ()
[12:21] * dignus (~jkooijman@t-x.dignus.nl) has joined #ceph
[12:21] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) Quit (Quit: Leaving.)
[12:24] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[12:24] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[12:26] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) has joined #ceph
[12:27] * sleinen1 (~Adium@2001:620:0:68::101) has joined #ceph
[12:32] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[12:34] * sleinen (~Adium@84-72-160-233.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[12:36] * LeaChim (~LeaChim@host81-159-253-189.range81-159.btcentralplus.com) has joined #ceph
[12:38] * dignus (~jkooijman@t-x.dignus.nl) Quit (Read error: Connection reset by peer)
[12:42] * Daviey__ (~DavieyOFT@bootie.daviey.com) Quit (Remote host closed the connection)
[12:46] * Daviey (~DavieyOFT@bootie.daviey.com) has joined #ceph
[12:51] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[12:53] * yanzheng (~zhyan@171.221.139.239) Quit ()
[12:56] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[13:03] * thomnico (~thomnico@2a01:e35:8b41:120:18ff:323:5106:f27) has joined #ceph
[13:03] * thomnico (~thomnico@2a01:e35:8b41:120:18ff:323:5106:f27) Quit (Remote host closed the connection)
[13:03] * thomnico (~thomnico@2a01:e35:8b41:120:18ff:323:5106:f27) has joined #ceph
[13:04] * steki (~steki@37.19.108.8) has joined #ceph
[13:09] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[13:10] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[13:19] <isodude> So odd, my cluster behaves badly, i.e. bad performance, I got degraded PGs (19 PGs around 42k objects), but it aint recovering. 0 o/s
[13:21] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) Quit (Remote host closed the connection)
[13:37] * lalatenduM (~lalatendu@122.172.214.130) Quit (Read error: Connection reset by peer)
[13:39] * rkdemon2 (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[13:43] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[13:44] * jtaguinerd (~jtaguiner@112.205.24.200) has joined #ceph
[13:56] * Sysadmin88 (~IceChat77@94.1.36.141) Quit (Quit: If you think nobody cares, try missing a few payments)
[13:56] * steki (~steki@37.19.108.8) Quit (Ping timeout: 480 seconds)
[14:00] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[14:02] * rotbeard (~redbeard@2a02:908:df19:5000:76f0:6dff:fe3b:994d) Quit (Quit: Verlassend)
[14:05] * KevinPerks (~Adium@2606:a000:80a1:1b00:6865:4077:9631:816e) has joined #ceph
[14:10] * linjan (~linjan@176.195.181.192) Quit (Ping timeout: 480 seconds)
[14:12] * danieljh (~daniel@0001b4e9.user.oftc.net) Quit (Quit: leaving)
[14:12] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[14:20] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[14:27] * lupu (~lupu@86.107.101.214) has joined #ceph
[14:32] * rkdemon2 (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[14:35] * linjan (~linjan@176.195.181.192) has joined #ceph
[14:36] * TMM (~hp@75.101.56.247) Quit (Ping timeout: 480 seconds)
[14:40] * jtaguinerd1 (~jtaguiner@112.205.24.200) has joined #ceph
[14:40] * jtaguinerd (~jtaguiner@112.205.24.200) Quit (Read error: Connection reset by peer)
[14:41] * jtaguinerd (~jtaguiner@203.215.116.66) has joined #ceph
[14:42] * tab_ (~oftc-webi@89-212-99-37.dynamic.t-2.net) has joined #ceph
[14:45] <tab_> hi how does ceph solve the full drive problem? For example, if I put an object using S3/Swift API (through radosgw) to PG which has 3 OSDs. Ceph finds out that disk1, to which it intended to write the object, is full. Will it put the object than to disk 2 or will return just an error to the user?
[14:45] * jtaguinerd (~jtaguiner@203.215.116.66) Quit ()
[14:46] * jtaguinerd1 (~jtaguiner@112.205.24.200) Quit (Read error: Connection reset by peer)
[14:57] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[15:00] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) has joined #ceph
[15:00] * linjan (~linjan@176.195.181.192) Quit (Read error: Operation timed out)
[15:05] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) has joined #ceph
[15:18] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) Quit (Remote host closed the connection)
[15:24] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[15:24] * diegows (~diegows@190.190.5.238) has joined #ceph
[15:28] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[15:29] * danieljh (~daniel@0001b4e9.user.oftc.net) Quit (Ping timeout: 480 seconds)
[15:31] * yanzheng (~zhyan@171.221.139.239) has joined #ceph
[15:36] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[15:53] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[16:00] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[16:05] * dneary (~dneary@96.237.180.105) has joined #ceph
[16:08] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[16:19] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) Quit (Remote host closed the connection)
[16:19] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) has joined #ceph
[16:20] * dmsimard_away is now known as dmsimard
[16:21] * dneary (~dneary@96.237.180.105) Quit (Ping timeout: 480 seconds)
[16:34] * rotbeard (~redbeard@2a02:908:df19:5000:76f0:6dff:fe3b:994d) has joined #ceph
[16:42] * dgbaley271 (~matt@c-98-245-167-2.hsd1.co.comcast.net) has joined #ceph
[16:42] * dmsimard is now known as dmsimard_away
[16:43] * yguang11 (~yguang11@vpn-nat.corp.tw1.yahoo.com) Quit (Ping timeout: 480 seconds)
[16:44] * nitti (~nitti@199.59.119.21) has joined #ceph
[16:44] * marrusl (~mark@2604:2000:60e3:8900:2dc1:c0bb:8e91:5b2d) Quit (Ping timeout: 480 seconds)
[16:46] * marrusl (~mark@2604:2000:60e3:8900:2dc1:c0bb:8e91:5b2d) has joined #ceph
[16:49] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) Quit (Remote host closed the connection)
[16:50] * lkoranda (~lkoranda@nat-pool-brq-t.redhat.com) has joined #ceph
[16:58] * JayJ (~jayj@ip-64-134-42-104.public.wayport.net) has joined #ceph
[16:58] * nitti (~nitti@199.59.119.21) Quit (Read error: Connection reset by peer)
[16:59] <JayJ> rkdemon: I'm for lookout for the same guide. no good resources yet
[16:59] <absynth__> tab_: if an OSD fills up, things become really, really messy in ceph
[17:00] * dmsimard_away is now known as dmsimard
[17:00] <absynth__> tab_: from a configurable percentage on (it's about 90%), you will see warnings in ceph -s and -w, and you will have to make sure the OSDs in question free up
[17:00] <absynth__> if an osd is full, (i think) writes are stalled, giving you a very unstable situation
[17:01] <tab_> absynth: so when primary OSD fills up within PG (let's say we have 3 OSDs within that PG), there is no automatic takeover of primary role by second OSD within PG?
[17:01] <JayJ> I need to help to figure out why "nova volume-detach" leaves the cinder volume in the "detaching" state for ever? Cinder backend is Ceph. There are no errors in any logs. rbd command shows the volume still in the Ceph pool. It appears that only way to detach is to terminate the instance. Any thoughts folks?
[17:02] <Kioob> per default, warning are at 85%, which stop backfilling too. And at 95% the whole cluster stop writes (you should really avoid that)
[17:02] <JayJ> Any issues with Openstack cinder with Ceph (firefly) backend that you guys know of?
[17:03] <tab_> absynth: also, does replication of the object always occour within the same PG? or could objects be replicated to other PGs?
[17:05] <absynth__> it's always inside the same PG
[17:05] <absynth__> that's the whole point of PGs
[17:07] * vbellur (~vijay@122.171.96.121) has joined #ceph
[17:10] <tab_> absynth: let's say that I have 4 OSDs within PG and 3replicas mode on that PG. Primary OSD get's filled up, other 3 disks are still free enough for writes. So CEPH still does not reasign primary role to secondary OSD, so that writing to that PG is possible?
[17:12] * dgurtner (~dgurtner@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[17:14] * yanzheng (~zhyan@171.221.139.239) Quit (Quit: This computer has gone to sleep)
[17:17] * nitti (~nitti@199.59.119.21) has joined #ceph
[17:21] * nitti (~nitti@199.59.119.21) Quit (Remote host closed the connection)
[17:21] * nitti (~nitti@199.59.119.21) has joined #ceph
[17:33] * JayJ (~jayj@ip-64-134-42-104.public.wayport.net) Quit (Quit: Computer has gone to sleep.)
[17:42] * diegows (~diegows@190.190.5.238) has joined #ceph
[17:42] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[17:42] * rotbeard (~redbeard@2a02:908:df19:5000:76f0:6dff:fe3b:994d) Quit (Quit: Verlassend)
[17:43] * danieagle (~Daniel@179.184.165.184.static.gvt.net.br) has joined #ceph
[17:43] * nitti (~nitti@199.59.119.21) Quit (Remote host closed the connection)
[17:49] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[17:52] * vbellur (~vijay@122.171.96.121) Quit (Quit: Leaving.)
[17:53] * KevinPerks (~Adium@2606:a000:80a1:1b00:6865:4077:9631:816e) Quit (Quit: Leaving.)
[17:53] * michalefty (~micha@ip25045ed2.dynamic.kabel-deutschland.de) has joined #ceph
[17:54] * nitti (~nitti@199.59.119.21) has joined #ceph
[17:57] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[17:59] * Sysadmin88 (~IceChat77@94.1.36.141) has joined #ceph
[18:06] * linjan (~linjan@176.195.181.192) has joined #ceph
[18:11] * michalefty (~micha@ip25045ed2.dynamic.kabel-deutschland.de) Quit (Quit: Leaving.)
[18:14] <flaf> Hi @all, I'm currently testing cephfs. My cluster has 3 nodes on Ubuntu Trusty (just VMs): node1=mon,mds(up),osd node2=mon,mds(standby),osd node3=mon,mds(standby),osd. And I have a node client which has mounted the cephfs from my cluster. If I poweroff node1 then it happens about 30-35 seconds during which the node client doesn't have any access to the cephfs (neither reading nor writing). And
[18:14] <flaf> after this time, the access is come back. Does this time of interruption (~ 30s) seem correct according to you?
[18:15] * nitti (~nitti@199.59.119.21) Quit (Remote host closed the connection)
[18:16] * steveeJ (~junky@HSI-KBW-085-216-022-246.hsi.kabelbw.de) Quit (Quit: Leaving)
[18:20] * steveeJ (~junky@HSI-KBW-085-216-022-246.hsi.kabelbw.de) has joined #ceph
[18:22] <Sysadmin88> checked for a configurable time variable for the standby to take control?
[18:23] <flaf> Sysadmin88: sorry I don't understand your question.
[18:23] * Nacer (~Nacer@2001:41d0:fe82:7200:adf3:e529:e925:e1f3) has joined #ceph
[18:24] <flaf> Ah... I have do no setting for that.
[18:24] <Sysadmin88> have you looked if there is a configuration setting for making that time shorter?
[18:25] <flaf> I just make a basic (and default) configuration. I don't known it there is a parameter for that. I will search in the mds configuration...
[18:25] <flaf> *if
[18:27] <flaf> should I search in mon settings or mds settings?
[18:27] * vbellur (~vijay@122.171.96.121) has joined #ceph
[18:28] * Nacer_ (~Nacer@2001:41d0:fe82:7200:587f:ffb2:e947:1340) has joined #ceph
[18:31] * JayJ (~jayj@pool-96-233-113-153.bstnma.fios.verizon.net) has joined #ceph
[18:32] * gregsfortytwo1 (~Adium@cpe-107-184-64-126.socal.res.rr.com) has joined #ceph
[18:34] * Nacer (~Nacer@2001:41d0:fe82:7200:adf3:e529:e925:e1f3) Quit (Ping timeout: 480 seconds)
[18:35] * linjan (~linjan@176.195.181.192) Quit (Ping timeout: 480 seconds)
[18:35] * thomnico (~thomnico@2a01:e35:8b41:120:18ff:323:5106:f27) Quit (Quit: Ex-Chat)
[18:40] <JayJ> Any of you using Ceph (firefly) with Cinder in Openstack environment?
[18:41] * dgbaley271 (~matt@c-98-245-167-2.hsd1.co.comcast.net) Quit (Remote host closed the connection)
[18:41] * darkfader (~floh@88.79.251.60) Quit (Read error: Operation timed out)
[18:44] <tab_> Does CEPH always sets OSDs within same PG to be in different availability zones (different nodes , racks) or is this in the user hand to set-up?
[18:47] * JayJ (~jayj@pool-96-233-113-153.bstnma.fios.verizon.net) Quit (Quit: Computer has gone to sleep.)
[18:48] * cmendes0101 (~cmendes01@pool-98-112-76-22.lsanca.fios.verizon.net) has joined #ceph
[18:50] <cmendes0101> I'm having trouble taking down an osd. stop osd says it is done but when I try to rm it says its still running. This is on a test setup. How can I just force remove it?
[18:52] * ccheng (~ccheng@c-50-165-131-154.hsd1.in.comcast.net) has joined #ceph
[18:54] * darkfader (~floh@88.79.251.60) has joined #ceph
[18:56] * JayJ (~jayj@pool-96-233-113-153.bstnma.fios.verizon.net) has joined #ceph
[19:01] * linjan (~linjan@176.195.181.192) has joined #ceph
[19:04] * JayJ (~jayj@pool-96-233-113-153.bstnma.fios.verizon.net) Quit (Quit: Computer has gone to sleep.)
[19:08] <flaf> cmendes0101: just kill the processes?
[19:08] * vbellur (~vijay@122.171.96.121) Quit (Read error: Connection reset by peer)
[19:09] * vbellur (~vijay@122.171.96.121) has joined #ceph
[19:12] * lupu (~lupu@86.107.101.214) Quit (Ping timeout: 480 seconds)
[19:12] * danieljh (~daniel@0001b4e9.user.oftc.net) Quit (Quit: leaving)
[19:14] * vbellur (~vijay@122.171.96.121) Quit ()
[19:23] <flaf> With cephfs, I can see bad perf when I try "ls -l /mnt/". Take 2.2s with /mnt which contains ~ 100 files.
[19:24] <flaf> Is it possible to improve this?
[19:36] * KevinPerks (~Adium@2606:a000:80a1:1b00:b861:8a5e:6a6f:893a) has joined #ceph
[19:38] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) has joined #ceph
[19:40] <JCL> cmendes0101: What version, what platform?
[19:41] <cmendes0101> ceph 0.80.5 on ubuntu 14.04
[19:42] <JCL> tab_: The CRUSH map will do what ever you want to do and will let you choose your failure domain but remember that a whole PG are always replicated by the same set of OSDs.
[19:46] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[19:47] <JCL> cmemndes0101: What do you have in /var/lib/ceph/osd/ceph-x (use pastebin to show) and what command do you issue?
[19:48] <tab_> JCL: thx. maybe also this quesiton. When one OSD within failed group fails, PG goes to degredated mode in case it could not guarantee the number of replicas, than CRUSH will just save the object to other PG?
[19:50] * sjm (~sjm@88.128.80.73) has joined #ceph
[19:56] <JCL> tab_: No the PGs hosted by this OSD will just be remapped to a different set of OSDs to make sure the number of replicas is fine. As far as I'm aware, if the number PGs remains unchanged (which would be true in the case you describe), the result of the hashing function will always place the object in the same PG. And then, CRUSH will map the PG to the available OSDs given the rules and the OSDs available
[20:00] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) has joined #ceph
[20:03] * sjm1 (~sjm@88.128.80.73) has joined #ceph
[20:04] <tab_> JCL: can 2 different PG with the same replication level have the same OSDs configured?
[20:04] * `10` (~10@69.169.91.14) has joined #ceph
[20:06] * sjm (~sjm@88.128.80.73) Quit (Ping timeout: 480 seconds)
[20:07] * `10_ (~10@69.169.91.14) has joined #ceph
[20:09] * `10__ (~10@69.169.91.14) has joined #ceph
[20:09] <JCL> tab_: Yes one OSd will host multiple PGs so if you issue a ceph pg dump command you can actually check PG mapping to OSDs on the fly and see that some PGs are replicated by the same OSD
[20:10] * `10___ (~10@69.169.91.14) has joined #ceph
[20:11] <JCL> tab_: Try ceph pg dump | grep acting[x,y,z] where x, y and z are 3 different OSD ids in separate failure domains and it will give you the list of the PGs replicated by the same set of OSDs
[20:11] * `10 (~10@69.169.91.14) Quit (Ping timeout: 480 seconds)
[20:11] <tab_> JCL: ok thx
[20:11] * `10__ (~10@69.169.91.14) Quit (Remote host closed the connection)
[20:11] * sjm1 (~sjm@88.128.80.73) Quit (Ping timeout: 480 seconds)
[20:13] * sjm (~sjm@88.128.80.73) has joined #ceph
[20:13] * `10` (~10@69.169.91.14) Quit (Remote host closed the connection)
[20:15] * `10_ (~10@69.169.91.14) Quit (Ping timeout: 480 seconds)
[20:36] * sjm (~sjm@88.128.80.73) has left #ceph
[20:41] <tab_> JCL: one more question. In case I have 3 racks, each in it's own failure domain, where each replica (1 out of 3) of data is placed. Let' say I have set all PGs to consits of three disks which are always in 3 different racks. How would CEPH react to power-failure of one rack. Would it start trying to make replicas to ohter two racks within different PGs just to guarantee 3-replicas (let's say I have min_size=3)? What when rack comes back online
[20:41] * dneary (~dneary@96.237.180.105) has joined #ceph
[20:42] <absynth__> yes, it would start rebalancing immediately
[20:42] <absynth__> if that rack comes back online after a rebalance, it's redundant
[20:43] <tab_> so if the rack comes back online , than it would place that replica back to previous disk so that disks are filled evenly?
[20:43] <absynth__> yeah
[20:44] <absynth__> so you'd be rebalancing twice
[20:44] <absynth__> seeing that rebalancing is a massive IO hog and terribly imperformant, that is not a desirable situation
[20:44] <absynth__> in practice, if you _know_ the rack is going to come back soon, you would set "noout" to avoid immediate rebalancing
[20:44] <tab_> in case , when I have min_size=2 than it is no problem?
[20:44] <absynth__> and then after the rack is back online, you'd only have a minor "backfill", i.e. writing replicas to that rack
[20:45] <absynth__> yes it is, because your rack would be part of a failure domain, still
[20:45] * Jeff10 (~oftc-webi@WE.ST.HMC.Edu) Quit (Quit: Page closed)
[20:45] * dneary (~dneary@96.237.180.105) Quit (Read error: Operation timed out)
[20:45] <JCL> tab_: From what I read in your message @ 11:41 - Be careful PGs are not disks
[20:45] * jamin (~jcollins@65-100-221-49.dia.static.qwest.net) has joined #ceph
[20:46] <tab_> JCL: yes PG are set of disks
[20:46] <jamin> anyone have thoughts on this: http://tracker.ceph.com/issues/9369
[20:46] <tab_> but disks could be defined within different nodes in different racks, right?
[20:47] <tab_> absynth: if rack is part of failure domain, than data is always replicated, no matter what min_size is set?
[20:48] <JCL> tab_ PGs are not disks. OSDs are devices hence disks. PGs are mapped to OSDs. And the OSDs will make sure a copy of each object in the PG has the appropriate number of replica on the appropriate number of OSD located where instructed to find them
[20:51] <JCL> tab_: min_size dictates how many replicas must be available for the OSDs to serve IO request to the PGs beloning to the pool.
[20:52] <JCL> tab_ So first on rack failure when the OSDs become down, the PGs handled by the OSDs in the rack will be remapped
[20:53] <JCL> tab_: And after 5 minutes, if the rack does not come back up, the OSDs will be marked 'out' and the recovery will kick in for the PGs hosted on the 'down' OSDs
[20:57] <tab_> JCL: what if after 1 day rack comes back online, what's with the data on OSD from failed rack - is it deleted for CEPH to use it again?
[21:00] * johntwilkins (~john@2601:9:4580:289:403d:8d2a:4c7:328c) has joined #ceph
[21:01] <JCL> tab_ Backfill will occur and data will be resynchronized between the original OSDs and the "backup" OSDs (the one that took over during the rack failure) that have the last version of each PGs.
[21:02] <Sysadmin88> i think where he's coming from is, does it sync the differences between the already present files or does it clean the files in the OSD and transfer the whole lot
[21:02] <jamin> tab, along with periodic scrubbing that will clear/free any redundant space/data
[21:02] <jamin> Sysadmin88, AFAIK, a mix of both, whatever is better
[21:02] <jamin> in terms of crush map balancing
[21:04] * dmsimard is now known as dmsimard_away
[21:04] <jamin> I've watched my cluster with the addition of new drives actively migrate data away from more heavily loaded OSDs to the new OSD to even overall utilization
[21:04] * JayJ (~jayj@pool-96-233-113-153.bstnma.fios.verizon.net) has joined #ceph
[21:04] <jamin> if my cluster wasn't currently impaired, I'd have nice graphs I could show you
[21:05] <Sysadmin88> what makes it impaired?
[21:05] <tab_> JCL: one more thing, how does CEPH choose the OSDs within remap situation - is this some predefined algoritm?
[21:05] <jamin> Sysadmin88, http://tracker.ceph.com/issues/9369
[21:05] <Sysadmin88> crush :)
[21:06] <jamin> tab_, yea, crush
[21:06] <tab_> ok. so it will choose best-effort way to set up new PG group of OSDs and correct crush map configuration file?
[21:06] <jamin> Sysadmin88, it's been like this for over a day. I'm trying to be patient, hoping it will work itself out as it seems to be trying to move things around and recover
[21:07] <jamin> tab_, it seems to do a very good job of it on its own, if you use ceph-deploy to turn up new OSDs
[21:07] <jamin> granted, I'm still a very new user
[21:09] * JayJ (~jayj@pool-96-233-113-153.bstnma.fios.verizon.net) Quit (Quit: Computer has gone to sleep.)
[21:14] * davidz (~Adium@cpe-23-242-12-23.socal.res.rr.com) Quit (Quit: Leaving.)
[21:15] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[21:15] <jamin> Sysadmin88, any ideas?
[21:20] <jamin> at this point I'd almost be willing to turn off all client use of the storage cluster if there was some way to have it focus 100% on recovery
[21:25] * rendar (~I@host215-182-dynamic.19-79-r.retail.telecomitalia.it) Quit (Read error: Operation timed out)
[21:30] * rendar (~I@host215-182-dynamic.19-79-r.retail.telecomitalia.it) has joined #ceph
[21:49] <jamin> sage: you around?
[21:52] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[21:53] * JayJ (~jayj@50-195-45-225-static.hfc.comcastbusiness.net) has joined #ceph
[21:56] <absynth__> jamin: what was your problem again? scrollback buffer too short here
[21:56] <jamin> absynth__: http://tracker.ceph.com/issues/9369
[21:56] <jamin> ceph-osd processes are terminating with ABRT
[21:57] <jamin> Ubuntu's upstart keeps standing them back up, but log files are growing fast, and cluster is not fully recovering
[21:57] <absynth__> ah...
[21:57] <absynth__> how many OSDs are affected?
[21:58] <absynth__> are they all on one node?
[21:58] <jamin> no
[21:58] <jamin> total of three nodes
[21:58] <jamin> 12 total OSDs (different numbers per node) same overall storage space per node
[21:58] <absynth__> how many total OSDs do you have?
[21:58] <jamin> and seems to be spread across all OSDs at different times
[21:59] <absynth__> oh
[21:59] <absynth__> that's bad
[21:59] <absynth__> if there wer just, say, 4 out of 30 osds flapping, you could just leave them down
[22:00] <absynth__> is that a cluster that is in production?
[22:00] <jamin> right now looks like 3 main flappers
[22:00] <absynth__> set osd noout, and try to kill them the hard way
[22:00] <jamin> 1 on one host and 2 on another
[22:01] <jamin> already got noout set to avoid their flapping causing undue degrading
[22:01] <jamin> guess I can try taking them down to see how much can recover
[22:01] <absynth__> i'd take them down, and see what happens them
[22:01] <absynth__> then, even
[22:02] <absynth__> if the cluster stabilizes, you can either let it recover or keep it like that during the night or whatever it is right now where you are
[22:02] <absynth__> i don't think you'll see a lot of inktank people here today and tomorrow
[22:02] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) has joined #ceph
[22:02] <jamin> ok, the three are down
[22:02] <absynth__> ceph -s says?
[22:03] <jamin> osdmap e38324: 12 osds: 9 up, 12 in
[22:03] <absynth__> how much degradation?
[22:03] <jamin> http://pastebin.com/KRve1CzQ
[22:03] <jamin> it's NOT pretty
[22:04] <absynth__> mh, the down objects are a bit worrying
[22:04] <absynth__> the rest is as expected, since you have 25% of your cluster down now
[22:04] <jamin> it's been complaining about a pg with 14 unfound objects
[22:05] <jamin> I'll monitor the OSD space, hopefully it doesn't fill my other OSDs completely trying to recover
[22:05] <jamin> these were three of the largest drives
[22:06] <jamin> IO on all VMs has pretty much ceased due to this
[22:08] * JayJ (~jayj@50-195-45-225-static.hfc.comcastbusiness.net) Quit (Quit: Computer has gone to sleep.)
[22:08] <absynth__> is this production or testing?
[22:09] <jamin> production
[22:09] <absynth__> meh, that sucks
[22:09] <absynth__> been there, felt that
[22:10] <jamin> but for some of the data I do have pre-ceph backups
[22:10] <absynth__> you won't lose data. that, i think, is the good news
[22:10] <jamin> is there a way to map PGs to RBDs? to know which RBD is impacted by which PGs?
[22:10] <absynth__> errm, yeah
[22:11] <absynth__> i don't remember though
[22:11] <jamin> been searching, haven't been able to find the right infor
[22:11] <jamin> info*
[22:11] <absynth__> ceph pg dump
[22:11] <absynth__> dumps your PGs
[22:12] <absynth__> not sure how to go from there, but there definitely is a very obscure and esoteric way
[22:12] <absynth__> try starting one of the downed OSDs and see if it stays up
[22:12] <absynth__> if it does, start the second
[22:13] <absynth__> and so on
[22:13] <absynth__> maybe it stabilizes
[22:13] <absynth__> i gotta go, laptop battery empty and no charger here
[22:13] <absynth__> sorry
[22:13] <jamin> absynth__: np, thanks for the help
[22:13] <jamin> the cluster is more stable now than it was, there are no "inactive" pgs
[22:13] <jamin> going to have to read up on all these states though
[22:15] <jamin> so, "down" means no IO, was afraid of that
[22:21] <jamin> sadly the OSD with most of the down PGs keeps flapping
[22:30] * JayJ (~jayj@50-195-45-225-static.hfc.comcastbusiness.net) has joined #ceph
[22:30] * linjan (~linjan@176.195.181.192) Quit (Ping timeout: 480 seconds)
[22:37] * eoin (~eoin@APoitiers-652-1-429-153.w83-193.abo.wanadoo.fr) Quit (Remote host closed the connection)
[22:45] * Meistarin (~Dont@0001c3c8.user.oftc.net) has joined #ceph
[22:56] * bkunal (~bkunal@113.193.136.56) has joined #ceph
[22:57] * JayJ (~jayj@50-195-45-225-static.hfc.comcastbusiness.net) Quit (Quit: Computer has gone to sleep.)
[22:59] * xarses (~andreww@c-76-126-112-92.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[22:59] * rotbeard (~redbeard@2a02:908:df19:5000:76f0:6dff:fe3b:994d) has joined #ceph
[23:08] * xarses (~andreww@c-76-126-112-92.hsd1.ca.comcast.net) has joined #ceph
[23:21] <steveeJ> jamin: could it be that your osd daemons hit the open-file-max limit?
[23:22] <steveeJ> i've had that once in a recovery situation. i had to adjust a configuration file to increase that limit
[23:26] * dignus (~jkooijman@t-x.dignus.nl) has joined #ceph
[23:35] <jamin> steveeJ: that is an interesting thought
[23:36] <steveeJ> also, I asked myself the same question with the pg -> pool mapping. i think it's just simpler than you would think. the pgid looks like <pool>.<id>
[23:37] <jamin> steveeJ: I'm looking to go further than pool though... I know which pool they are in... I'm only using one
[23:37] <steveeJ> well you rather want to know pg -> rbd image, but you know pool -> rbd
[23:37] <steveeJ> oh, so what is not clear?
[23:37] <jamin> which specific rbd image is impacted by the missing pgs
[23:37] * zerick (~eocrospom@190.118.30.195) has joined #ceph
[23:38] <steveeJ> ah i see
[23:38] <jamin> if it's one I'm comfortable with restoring from a pre-ceph backup, I'd be wiling to just lose the pgs to recover the cluster
[23:39] <steveeJ> if your situation is not that critical right now, I'd use it as an exercise to get it running
[23:39] <steveeJ> if you know the PGs are somewhere and your problem is not clear, don't just skip it and hope it's not going to happen again
[23:39] <jamin> it's not critical, it's my home production cluster, but it's annoying that all my VMs are offline
[23:40] <jamin> steveeJ: agreed, I know where most of the pgs are
[23:40] <jamin> there is one set that is concerning though
[23:41] <jamin> looks like the limits for the ceph-osd process is:
[23:41] <jamin> Max open files 32768 32768 files
[23:41] <steveeJ> that limit is probably going to be enough
[23:42] <jamin> personally, I think it's a specific pg that it's trying to restore that is causing the ABRT, but I'm not sure how to read the dump
[23:43] <jamin> which is also why I'm looking for pg -> rbd_volume mapping
[23:43] <jamin> is there a way to tell ceph to stop trying to recover a specific pg "for now"?
[23:43] <jamin> basically, deprioritize it
[23:44] <steveeJ> mh, i don't know if you can revert the "lost" state
[23:44] <steveeJ> but there is a lost state, in case you didn't know
[23:44] <jamin> for pg or osd?
[23:44] <jamin> I know of the osd lost state, but that's for all data on that OSD, yes?
[23:45] <steveeJ> http://ceph.com/docs/master/rados/operations/control/#placement-group-subsystem
[23:45] * rkdemon (~rkdemon@pool-71-244-62-208.dllstx.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[23:45] <steveeJ> that's what i meant: ceph pg {pgid} mark_unfound_lost revert|delete
[23:47] <jamin> tried that on one pg already and it refused to do it
[23:47] <jamin> said that not all volumes had been checked
[23:47] <jamin> was trying the revert
[23:49] <jamin> based on absynth__'s suggestion earlier I took the 3 main problem OSDs down
[23:49] <steveeJ> can't help you with experience here. i haven't yet and hopefully will not need that
[23:49] <jamin> and the cluster looks like it's recovered as far as it can without them
[23:50] <jamin> but bringing the one most needed back online just starts flapping again
[23:51] <jamin> and it's responsible for 40 of the down pgs from what "ceph pg dump" says
[23:52] * zerick (~eocrospom@190.118.30.195) Quit (Ping timeout: 480 seconds)
[23:52] <jamin> sadly there are three that don't appear to have a known OSD for them
[23:54] <jamin> any experience reading the ceph-osd log dumps?
[23:57] <jamin> steveeJ: 30 seconds after the OSD process starts it ABRTS

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.