#ceph IRC Log

Index

IRC Log for 2012-10-27

Timestamps are in GMT/BST.

[0:00] <rlr219> we can try to fix it for maybe a couple of more days, but at some point maybe we should just start over. I really only have 4 VMs running and I have good exports from a couple of weeks ago, as i haven't exported since we hit this issue.
[0:00] <rlr219> obviously, if we can just fix these 3 OSDs that wiould be better.
[0:01] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:02] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[0:04] <rlr219> do you need the big log?
[0:07] <rlr219> cheers.
[0:08] * rlr219 (43c87e04@ircip4.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[0:09] * nwatkins1 (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[0:09] * noahdesu (~Adium@soenat3.cse.ucsc.edu) Quit (Read error: Connection reset by peer)
[0:10] * nwatkins2 (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[0:10] * nwatkins1 (~Adium@soenat3.cse.ucsc.edu) Quit (Read error: Connection reset by peer)
[0:19] * rweeks (~rweeks@12.25.190.226) Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[0:25] * nwatkins1 (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[0:25] * nwatkins2 (~Adium@soenat3.cse.ucsc.edu) Quit (Read error: Connection reset by peer)
[0:32] * nhmlap (~nhm@38.122.20.226) Quit (Ping timeout: 480 seconds)
[0:32] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) Quit (Quit: \o/)
[0:36] * justinwarner (~Thunderbi@130.108.232.145) Quit (Quit: justinwarner)
[0:36] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[0:38] * nwatkins1 (~Adium@soenat3.cse.ucsc.edu) Quit (Quit: Leaving.)
[0:57] * rweeks (~rweeks@12.25.190.226) has joined #ceph
[1:01] <elder> gmail really seems to be throttling me or something. Is it really slow for anyone else?
[1:04] * miroslav (~miroslav@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[1:15] * danieagle (~Daniel@177.133.174.226) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[1:21] <sagewk> joshd, dmick: wip-oc-neg is surviving my testing.. want to take a look?
[1:26] <joshd> sure
[1:30] * nhmlap (~nhm@216.3.101.62) has joined #ceph
[1:40] * sjust (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) Quit (Ping timeout: 480 seconds)
[1:48] * Tamil (~Adium@38.122.20.226) has left #ceph
[1:49] * lofejndif (~lsqavnbok@83TAAB1A6.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[1:51] <dmick> osdc/Blinker.h is just totally dead, right?
[1:53] <gregaf> I...what?
[1:53] <gregaf> what is this file in my tree?
[1:54] <dmick> uh, yeah, what I said
[1:56] <gregaf> I'm going to call it dead, yes
[2:20] * rweeks (~rweeks@12.25.190.226) Quit (Quit: Computer has gone to sleep.)
[2:28] * gregaf (~Adium@2607:f298:a:607:20e0:1784:3237:9cd4) Quit (Quit: Leaving.)
[2:29] * sagelap1 (~sage@100.sub-70-197-145.myvzw.com) has joined #ceph
[2:30] * nhmlap (~nhm@216.3.101.62) Quit (Read error: No route to host)
[2:31] * sagelap (~sage@2607:f298:a:607:aca0:7479:696a:6818) Quit (Ping timeout: 480 seconds)
[2:33] * nhmlap (~nhm@216.3.101.62) has joined #ceph
[2:45] * anna1 (~Adium@173.247.207.74) has left #ceph
[2:59] * Cube (~Cube@12.248.40.138) Quit (Quit: Leaving.)
[3:11] * nhmlap (~nhm@216.3.101.62) Quit (Ping timeout: 480 seconds)
[3:22] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[3:28] * noob2 (47f46f24@ircip1.mibbit.com) has joined #ceph
[3:29] <noob2> are people generally seeing better performance with ceph using flash backed raid controllers or having the journal on ssd?
[3:33] * sagelap1 (~sage@100.sub-70-197-145.myvzw.com) Quit (Ping timeout: 480 seconds)
[3:40] * noob2 (47f46f24@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[3:58] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) Quit (Quit: Leaving)
[4:00] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) Quit (Quit: Leseb)
[4:32] * phil_ (~user@87.117.192.173) Quit (Remote host closed the connection)
[5:03] * adjohn (~adjohn@69.170.166.146) Quit (Quit: adjohn)
[5:07] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[5:15] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[5:35] * erlkonig (~alex@yggdrasil.talisman.org) has joined #ceph
[5:40] <erlkonig> Anyone using CephFS who wouldn't mind commenting on how viable it is currently?
[5:41] <nwl> erlkonig: it's totally usable but we are saying it is not ready for 'production' use as there are some elements we want to QA more and refine
[5:41] <nwl> erlkonig: you can consider it ready as a 'tech preview'
[5:42] <erlkonig> Would you move your home directory into it? ;-)
[5:43] <nwl> erlkonig: depends what your home directory contains :)
[5:44] <erlkonig> I'm considering trying it out to bind about 1 TiB together on my home systems. Essentially I'm wondering if I'd probably need to restore from backup in a few months.
[5:44] <erlkonig> It'd be really handy to be able to have a node in the cloud as part of the same fs.
[5:46] <nwl> so CephFS split across high latency networks is not on the immediate roadmap except in a DR architecture
[5:46] <erlkonig> On the other hand, if the team is still ironing out kernel panics or file system corruption I should wait. Low grade performance issues I can handle, though
[5:46] <nwl> ie you can't have two nodes at home and one in AWS and right to any of them
[5:46] <erlkonig> Ah
[5:46] <erlkonig> Darn :-)
[5:47] <nwl> we are looking at having it so you can have the 'cloud' nodes as a failover cluster
[5:47] <erlkonig> Hmm.. Are than any filesystems that are working well currently for bridging two geographically-separated sites?
[5:48] <nwl> None that I can personally recommend
[5:49] <erlkonig> It does sound like something that would fit into a future version of Ceph, but I don't know if it's on the roadmap at all
[5:49] <nwl> It is on the roadmap but not the near term one
[5:49] <erlkonig> Okay. Thanks nw1, that's exactly the info I needed.
[5:50] <erlkonig> (but I'll check back on Ceph later to see if CephFS heads where I'm thinking)
[5:53] <erlkonig> Was "one node" the no-write ban? What about having a full cluster at both sites - failover only?
[6:02] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[6:06] * rweeks (~rweeks@12.25.190.226) has joined #ceph
[6:15] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[6:16] * erlkonig (~alex@yggdrasil.talisman.org) has left #ceph
[6:22] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[6:23] * dmick (~dmick@2607:f298:a:607:dd60:ae90:5409:88f1) Quit (Quit: Leaving.)
[6:39] * deepsa_ (~deepsa@122.172.33.114) has joined #ceph
[6:40] * deepsa (~deepsa@122.172.13.9) Quit (Ping timeout: 480 seconds)
[6:40] * deepsa_ is now known as deepsa
[6:57] * Yann__ (~Yann@did75-15-88-160-187-237.fbx.proxad.net) has joined #ceph
[7:01] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[7:04] * kYann (~Yann@did75-15-88-160-187-237.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[7:05] * nhmlap (~nhm@184-97-251-146.mpls.qwest.net) has joined #ceph
[7:07] * rweeks (~rweeks@12.25.190.226) Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[7:17] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[7:36] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[8:15] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[8:17] * spaceman139642 (l@89.184.139.88) has joined #ceph
[8:19] * spaceman-39642 (l@89.184.139.88) Quit (Read error: Connection reset by peer)
[8:27] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[8:47] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[8:47] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) Quit (Remote host closed the connection)
[8:48] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) has joined #ceph
[8:50] * LarsFronius (~LarsFroni@2a02:8108:3c0:79:f4e7:702:faad:204c) has joined #ceph
[8:54] * LarsFronius (~LarsFroni@2a02:8108:3c0:79:f4e7:702:faad:204c) Quit ()
[8:56] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) Quit (Ping timeout: 480 seconds)
[8:56] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[9:18] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) has joined #ceph
[9:30] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) Quit (Ping timeout: 480 seconds)
[9:56] * Q310 (~qgrasso@ip-121-0-1-110.static.dsl.onqcomms.net) Quit (Ping timeout: 480 seconds)
[10:02] * vikey (~Vikey@218.206.243.163) has joined #ceph
[10:03] <vikey> hello
[10:05] * vikey (~Vikey@218.206.243.163) Quit ()
[10:10] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[10:22] * jakku (~jakku@ac232088.dynamic.ppp.asahi-net.or.jp) has joined #ceph
[10:47] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[11:02] * loicd (~loic@magenta.dachary.org) has joined #ceph
[11:45] * stass (stas@ssh.deglitch.com) Quit (Read error: Connection reset by peer)
[11:49] * stass (stas@ssh.deglitch.com) has joined #ceph
[12:13] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) has joined #ceph
[12:14] <alan_> hi
[12:16] <alan_> i find problem, why "HEALTH_WARN 72 pgs stale; 72 pgs stuck stale", but i don't know how fix it.
[12:19] <alan_> all 72 pg from odd 8,6. I have problem with this disks, and i understand, i have not information from pgs. But how i can flush problem without formatting all osd?
[12:48] * madkiss (~madkiss@178.188.60.118) Quit (Quit: Leaving.)
[13:04] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[13:04] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit ()
[13:09] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[13:23] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[13:26] * tryggvil (~tryggvil@gw-vm-gestir.hir.is) has joined #ceph
[13:58] * Cube1 (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[14:07] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) has joined #ceph
[14:16] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[14:44] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) Quit (Quit: alan_)
[14:46] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) has joined #ceph
[14:53] * tryggvil (~tryggvil@gw-vm-gestir.hir.is) Quit (Quit: tryggvil)
[14:59] * Cube1 (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[15:09] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[15:10] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[15:14] * lofejndif (~lsqavnbok@82VAAHHXR.tor-irc.dnsbl.oftc.net) has joined #ceph
[15:20] * stxShadow (~Jens@ip-178-203-169-190.unitymediagroup.de) has joined #ceph
[15:51] * tryggvil (~tryggvil@gw-vm-gestir.hir.is) has joined #ceph
[15:53] * stxShadow (~Jens@ip-178-203-169-190.unitymediagroup.de) has left #ceph
[15:58] * lofejndif (~lsqavnbok@82VAAHHXR.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[16:05] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[16:34] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) has joined #ceph
[16:36] * coredumb (~coredumb@ns.coredumb.net) Quit (Quit: WeeChat 0.3.8)
[16:41] * Kioob (~kioob@luuna.daevel.fr) has joined #ceph
[16:42] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) Quit (Ping timeout: 480 seconds)
[16:55] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[16:55] * loicd (~loic@magenta.dachary.org) has joined #ceph
[16:56] * long (~chatzilla@118.186.58.111) has joined #ceph
[16:59] * lofejndif (~lsqavnbok@659AABJCM.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:35] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) has joined #ceph
[17:37] * madkiss (~madkiss@178.188.60.118) has joined #ceph
[17:42] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) has joined #ceph
[17:54] * psomas (~psomas@inferno.cc.ece.ntua.gr) has joined #ceph
[17:56] * lofejndif (~lsqavnbok@659AABJCM.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[18:03] * scalability-junk (~stp@188-193-208-44-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[18:11] * LarsFronius (~LarsFroni@2a02:8108:3c0:79:ed77:6866:bae9:3733) has joined #ceph
[18:12] * danieagle (~Daniel@177.99.135.20) has joined #ceph
[18:25] * LarsFronius_ (~LarsFroni@95-91-242-157-dynip.superkabel.de) has joined #ceph
[18:27] * LarsFronius (~LarsFroni@2a02:8108:3c0:79:ed77:6866:bae9:3733) Quit (Ping timeout: 480 seconds)
[18:27] * LarsFronius_ is now known as LarsFronius
[18:48] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) Quit (Quit: alan_)
[18:50] * madkiss (~madkiss@178.188.60.118) Quit (Quit: Leaving.)
[18:54] * tryggvil (~tryggvil@gw-vm-gestir.hir.is) Quit (Quit: tryggvil)
[19:03] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[19:11] * tryggvil (~tryggvil@163-60-19-178.xdsl.simafelagid.is) has joined #ceph
[19:20] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[19:22] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:25] * danieagle (~Daniel@177.99.135.20) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[19:26] * psomas (~psomas@inferno.cc.ece.ntua.gr) Quit (Ping timeout: 480 seconds)
[19:30] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[19:41] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[19:49] * long (~chatzilla@118.186.58.111) Quit (Quit: ChatZilla 0.9.89 [Firefox 16.0.1/20121010144125])
[20:11] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[20:16] * Lennie_ (~leen@524A9CD5.cm-4-3c.dynamic.ziggo.nl) has joined #ceph
[20:18] <Lennie_> Hi, at work we've been thinking of using Ceph for our needs, maybe even Cephfs in the future when it is ready.
[20:18] <Lennie_> But today I read this article: http://joyent.com/blog/network-storage-in-the-cloud-delicious-but-deadly
[20:18] <Lennie_> And now I'm thinking he's kinda right
[20:19] <Lennie_> Any thoughts on that ?
[20:21] <Lennie_> summary: when you store data on the network you increase complexity and make it harder to diagnose problems; with a possibility of cascading failures
[20:22] <Lennie_> Not that ceph is extremely complex, that is why I think it is very interresting. :-)
[20:36] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) has joined #ceph
[20:36] <joao> Lennie_, I guess it all comes down to how much can you benefit from such setup vs your willingness to deal with potential issues
[20:37] <Lennie_> yes, I understood it's about choices, striking the balance that fits the task
[20:38] <joao> just finished reading the article, and imo there were other issues that would worth be mentioned rather than the noatime bit
[20:38] <joao> for instance, network bottlenecks
[20:39] <Lennie_> I think the point of the article was that they didn't find the bottlenecks because they didn't know how to analyze it with all the different activity in the multi-tenant environment
[20:39] <joao> or other performance stuff that I'm not familiar with but which I'm sure exists :p
[20:40] <Lennie_> anyway, I did have a real Ceph question also:
[20:40] <Lennie_> does a client have the crush-map and pg-information ?
[20:40] <joao> yes
[20:41] <joao> but doesn't need to have it beforehand
[20:41] <joao> the monitors will share it with the client
[20:41] * tryggvil (~tryggvil@163-60-19-178.xdsl.simafelagid.is) Quit (Remote host closed the connection)
[20:42] <joao> and by client I mean one of the ceph components (libs, rgw, cephfs client)
[20:42] <Lennie_> does that mean the client could be made smart enough to choose the OSD closest to him (network wise or whatever the policy dictates)
[20:42] <Lennie_> forgot the question mark at the end ;-)
[20:43] <joao> I don't think the client has much option when it comes to that
[20:43] <joao> the client will always deal with the osd that has the primary replica of the pg the client is accessing
[20:43] <Lennie_> ok
[20:44] <Lennie_> I was just trying to get a feel of how Ceph works and how certain details are handled.
[20:44] <Lennie_> thanks for the information
[20:45] <joao> you're most welcome :)
[20:52] <Lennie_> OK, I've got an other question: is there a way to add more redundancy on the network side. For example can a OSD have more than one IP-address which clients, monitoring daemons and so on know how to contact ?
[20:58] <alan_> hi joao can you help me? i find problem, but i don't know how fix it. As you remember my ceph health have warning "72 pgs stale; 72 pgs stuck stale". Because all 72 pg on OSD 6 and 8, information from 6 and 8 was lost.
[20:59] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[20:59] <joao> Lennie_, I'm not sure if this answers your question, but iirc you can have an IP for clients and another for the backend operations
[21:00] <Lennie_> joao: I did see that in the documentation, I guess the answer is: the 2 networks is it (currently)
[21:00] <alan_> now I chenge replication from 2 to 3, bus i don't know how flush warning without destroy all.
[21:00] <joao> alan_, I'm not sure how to help you with that
[21:02] <joao> and I don't think there's anyone online today to help you with that either
[21:02] <alan_> now i can destroy all, but if i have some problem on production… what i to do?
[21:02] <alan_> can you show me nickname who can help me?
[21:03] <joao> well, sjustlaptop would probably the one I would talk to if it was me, but I don't know if he's paying attention to IRC today :)
[21:04] <sjustlaptop> for the moment, anyway
[21:04] <sjustlaptop> alan_: what's going on?
[21:04] <joao> sjustlaptop, he has stale pgs
[21:04] <joao> <alan_> hi joao can you help me? i find problem, but i don't know how fix it. As you remember my ceph health have warning "72 pgs stale; 72 pgs stuck stale". Because all 72 pg on OSD 6 and 8, information from 6 and 8 was lost.
[21:05] <sjustlaptop> can you post the output of ceph -s and ceph osd dump
[21:05] <alan_> HEALTH_WARN 72 pgs stale; 72 pgs stuck stale
[21:05] <sjustlaptop> alan_: not just the health line
[21:06] <alan_> yes, one moment, but now i change replication from 2 to 3 and I have degraded storage.
[21:06] <sjustlaptop> ok
[21:06] <alan_> http://pastebin.com/Wk2ekgzs
[21:07] <sjustlaptop> ok, can you post ceph pg dump?
[21:08] <sjustlaptop> so to get the sequence of events right, you changed the replication from 2 to 3
[21:08] <alan_> yes, here http://pastebin.com/DwBZAipc before i change replication level
[21:08] <sjustlaptop> and then you have stale pgs?
[21:08] <sjustlaptop> or were the stale pgs there prior to changing the replication level?
[21:08] <alan_> yes, before i change.
[21:08] <sjustlaptop> I see, odd
[21:09] <sjustlaptop> ok, looks like I need ceph osd tree and ceph pg dump
[21:09] <alan_> tree http://pastebin.com/YPpG3dys
[21:10] * tryggvil (~tryggvil@15-80-126-149.ftth.simafelagid.is) has joined #ceph
[21:10] <alan_> pg dump http://pastebin.com/0KUKpPfy
[21:11] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Read error: Connection reset by peer)
[21:11] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[21:11] <sjustlaptop> sorry, laptop crashed
[21:11] <alan_> tree http://pastebin.com/YPpG3dys , pg dump http://pastebin.com/0KUKpPfy
[21:11] <sjustlaptop> thanks
[21:13] <alan_> now problem - a can not mount cephfs, i think because i have warning.
[21:13] * rweeks (~rweeks@12.25.190.226) has joined #ceph
[21:14] <sjustlaptop> can you post the output of 'ceph pg 2.22 query'
[21:14] <sjustlaptop> ?
[21:14] <alan_> i try it, no result
[21:14] <alan_> i don't have pgid 2.22
[21:15] <sjustlaptop> what error output did it give you?
[21:15] <alan_> "i don't have pgid 2.22"
[21:15] <alan_> only
[21:15] * tziOm (~bjornar@ti0099a340-dhcp0778.bb.online.no) has joined #ceph
[21:15] <sjustlaptop> ok, how about 'ceph map 2.22'
[21:16] <alan_> unrecognized command
[21:16] <alan_> osdmap e2767 pg 2.22 (2.22) -> up [6,8] acting [6,8]
[21:17] <alan_> i have crash 6,8. I think its problem.
[21:17] <sjustlaptop> sorry?
[21:17] <sjustlaptop> so osd daemons 6 and 8 are dead?
[21:17] <sjustlaptop> that would explain it
[21:17] <alan_> now not, i replace HDD
[21:17] <alan_> yes, but how i can fix it?
[21:17] <sjustlaptop> ah
[21:18] <sjustlaptop> you had replication level 2 at the time?
[21:18] <sjustlaptop> did you loose both 6 and 8 close together?
[21:18] <alan_> yes
[21:18] * rweeks (~rweeks@12.25.190.226) Quit ()
[21:18] <sjustlaptop> ah
[21:18] <sjustlaptop> those 72 pgs are the ones which were mapped to 6 and 8
[21:18] <sjustlaptop> so you lost data
[21:19] <alan_> yes, but i have other data.
[21:19] <sjustlaptop> sorry?
[21:19] <alan_> a have loss all data from storage?
[21:19] <sjustlaptop> no, just the data from those 72 pgs
[21:19] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) has joined #ceph
[21:19] <alan_> ok, how i can flush warning and mount cephfs?
[21:20] <sjustlaptop> we can remake the pgs, but you will have lost any objects stored there
[21:20] <sjustlaptop> and that has probably corrupted cephfs
[21:20] <alan_> only 72 objects?
[21:20] <sjustlaptop> 72 pgs
[21:20] <alan_> how i can remarke?
[21:21] <sjustlaptop> looks like a few 10,000 objects
[21:21] * Kioob (~kioob@luuna.daevel.fr) Quit (Ping timeout: 480 seconds)
[21:21] <alan_> no problem, here my test server
[21:21] <alan_> how i can do it?
[21:21] <sjustlaptop> looking it up
[21:22] <alan_> al osd up
[21:22] <alan_> all
[21:23] <eccojoke> I'm looking at exporting ceph rbd's using two iscsi "bridge" machines to 3-4 vm hosts (esxi), each running 10 virtual machines. I'm using the argonaut release. am i going to run in to locking issues?
[21:23] <eccojoke> what i am seeing so far is latency spikes upwards 15 seconds from time to time
[21:24] <sjustlaptop> for each stale pg: 'ceph pg force_create_pg <pgid>'
[21:24] <alan_> have i wait for replication ends?
[21:25] <sjustlaptop> that wouldn't hurt
[21:25] <sjustlaptop> it probably doesn't matter
[21:25] <sjustlaptop> though
[21:26] <alan_> i can not mount cephfs, this may be due to the fact that I have a warning?
[21:27] <sjustlaptop> cephfs stores it's data in the osds
[21:27] <sjustlaptop> the osds store their data in pgs
[21:27] <sjustlaptop> 72 of your pgs are missing at the moment
[21:27] <alan_> now 71 :) i try force_create
[21:27] <sjustlaptop> cool
[21:27] <sjustlaptop> did it work?
[21:27] <alan_> one moment, i force all gps
[21:27] <sjustlaptop> ok
[21:28] <sjustlaptop> I think that should suffice
[21:30] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) has joined #ceph
[21:30] <gregaf> the MDS is not going to be happy at losing data
[21:30] <gregaf> I suspect manual repair is going to be required — if it didn't need the data it wouldn't have gotten stuck looking for it
[21:30] <alan_> sjustlaptop: where i can find all command, like force_create_pg ?
[21:31] <alan_> can you post link ti wiki
[21:31] <sjustlaptop> I don't think that one's documented yet
[21:31] <gregaf> eccojoke: the bridge machines are using different images, right?
[21:31] <gregaf> so you're just worried because of the latency spikes?
[21:34] <alan_> sjustlaptop: health HEALTH_WARN 335 pgs backfill; 334 pgs degraded; 342 pgs recovering; 10 pgs stuck inactive; 352 pgs stuck unclean; recovery 683493/3276689 degraded (20.859%); 1 near full osd(s)
[21:34] <alan_> what about "10 pgs stuck inactive" ?
[21:34] <sjustlaptop> what is the rest of ceph -s?
[21:35] <sjustlaptop> alan_: for the record, this won't recover the data, it's just telling the cluster that the data is gone and not coming back
[21:35] <alan_> http://pastebin.com/PHhXeFaZ
[21:37] <gregaf> those 10 are still creating
[21:37] * Cube1 (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[21:37] <gregaf> well, recreating, really
[21:37] <sjustlaptop> that might take a minute or two to go away
[21:38] <alan_> ok, i will be wait.
[21:38] <sjustlaptop> can you pick one of the "creating" pgs and do 'ceph pg <pgid> query'
[21:38] <sjustlaptop> ?
[21:38] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[21:39] <alan_> http://pastebin.com/jtgBC7mC
[21:39] <sjustlaptop> I mean, do ceph pg dump, find one still "creating", and do pg query on that one
[21:40] <sjustlaptop> 2.22 appears to have recreated successfully
[21:41] <alan_> all recreated successfully.
[21:41] * danieagle (~Daniel@177.97.248.22) has joined #ceph
[21:42] <alan_> but i can not mount cephfs, i mount write "mount error 5 = Input/output error"
[21:42] <eccojoke> gregaf, no - they are two cause of multipathing ,they are exporting the same rbd
[21:42] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) has left #ceph
[21:42] <sjustlaptop> alan_: yeah, cephfs is missing metadata
[21:42] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) has joined #ceph
[21:42] <sjustlaptop> alan_: this would one day be the time to use cephfs fsck
[21:42] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) has left #ceph
[21:43] <sjustlaptop> but we don't have one yet
[21:43] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) has joined #ceph
[21:43] <alan_> :)
[21:43] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) Quit (Quit: L?mnar)
[21:44] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[21:44] * ecco (~ecco@h94n3-lk-d2.ias.bredband.telia.com) has joined #ceph
[21:44] * ecco is now known as eccojoke
[21:45] <alan_> What do you suggest?
[21:45] <alan_> Recreate all?
[21:45] <eccojoke> but yes, my main worry is the latency spikes
[21:46] <gregaf> eccojoke: okay, so RBD doesn't do coherency across multiple mounters — you'll need something else to ensure they aren't stepping on each other's toes...
[21:46] <gregaf> where are you seeing latency spikes? from the clients which are talking to the exports when accessing the disk?
[21:47] <eccojoke> ok, so that basically brings me back to single mount point / point of failure, which was the reason i was looking at ceph/rbd..
[21:47] <eccojoke> yes, latency spikes show up in esxi performance logs
[21:48] <Lennie_> eccojoke: I'm no ceph expert, but I do have a question: do you have 2 iscsi "bridge" machines because of availability or do you need loadbalancing as well ?
[21:48] <gregaf> eccojoke: well, it's designed to be used by eg KVM which can mount it directly — if you're re-exporting you'll have to figure out something else
[21:49] <eccojoke> lennie, HA, don't need the load balancing for now
[21:49] <gregaf> I'm not sure the details of your software in terms of its coherency management, maybe it does that already?
[21:49] <eccojoke> gregaf, yes, i understand that it isn't made for esxi, found a guide on google saying suggesting dual iscsi, so figured id give it a go
[21:49] <gregaf> anyway, if you have more data about the latency spikes we can help you track them down
[21:50] <eccojoke> http://ceph.com/wiki/ISCSI
[21:50] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[21:50] <eccojoke> maybe i misunderstood that, or its outdated
[21:50] <Lennie_> eccojoke: maybe you can do some kind of iscsi failover setup ? where only one "bridge" machine is talking to ceph rbd ?
[21:50] <gregaf> I'm not very familiar with ESXi but have you isolated it enough to be sure it's the RBD mounts and not the exporters which are laggy?
[21:50] <eccojoke> don't really have any data yet, figured id go ask here if its expected
[21:51] <eccojoke> could very well be the exporters, as i said - haven't really gathered any data as of yet.
[21:51] <gregaf> well, that's what you'd want to figure out ;)
[21:51] <gregaf> I'm afraid I'm not very familiar with those systems
[21:52] <alan_> sjustlaptop: What do you suggest? Recreate all?
[21:52] <gregaf> perhaps wido when he's around, looks like he wrote that iSCSI export wiki page
[21:52] <eccojoke> yes, but if using multiple iscsi "bridges" isn't possible, it doesn't really matter where the latency is
[21:52] <eccojoke> it just won't work as i thought, so back to drawing board i guess
[21:52] <gregaf> well, I'm sure it's possible to set up in a safe way, just from your description I'm not sure if it was safe or not
[21:53] <gregaf> so you've kernel-mounted RBD on your iSCSI targets?
[21:53] <eccojoke> yes
[21:54] <eccojoke> and then exported /dev/rbdX as iscsi targets on both machines
[21:54] <gregaf> and you're using the DirectIO option it mentions and not trying to access it from multiple initiators?
[21:54] <gregaf> (at the same time, I mean)
[21:54] <eccojoke> i might be
[21:54] <eccojoke> as i mapped the iscsi target to 4 initiators (esxi hosts)
[21:54] <gregaf> if that's the case I believe it's fine, but we get all sorts in here so when people talk about multiple mounters I get nervous
[21:55] <eccojoke> e.g. I've made a 500gb rbi, kernel mounted it on iscsi001 and iscsi002, exported as lun0 to esxi001-004
[21:55] <eccojoke> s/rbi/rbd
[21:56] <gregaf> one quick check you could try to see if the latency spikes are coming out of RADOS is to go look in the data directory of one of your monitors, where you will see a log file (I forget the exact name) and see if it has any warnings about slow operations
[21:56] <gregaf> if it does then that's the cause and we can dig into why the ops are slow
[21:56] <gregaf> if not, I think it's the iSCSI targets misbehaving
[21:56] <eccojoke> will do that tomorrow, will the actual log message be "slow operations", or do i need to look for timings?
[21:56] <gregaf> well, if your upper layers are themselves behaving then it's fine (ie, if this is a normal VMWare configuration)
[21:56] <alan_> Thank you guys. You've been very helpful.
[21:57] <eccojoke> yes, no tweaks on the vmware
[21:57] <gregaf> eccojoke: you don't need to look at timings, just grep for "slow"
[21:57] <eccojoke> cheers, will give it another go. ill report back
[21:57] <gregaf> sjustlaptop: actually, is it "slow" or "delayed"?
[21:57] <gregaf> I don't have one on hand to check with
[21:57] <eccojoke> ill grep for both
[21:58] <eccojoke> thanks very much for now!
[21:58] <gregaf> but I do have the source code, dir
[21:58] <gregaf> *dur
[21:58] <gregaf> "slow request"
[21:59] <gregaf> np
[22:02] * alan_ (~alan@ctv-95-173-34-17.vinita.lt) Quit (Quit: alan_)
[22:05] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[22:06] <Lennie_> have a good weekend
[22:06] <Lennie_> thanks for the info
[22:06] * Lennie_ (~leen@524A9CD5.cm-4-3c.dynamic.ziggo.nl) Quit (Quit: bye)
[22:15] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[22:28] * Cube1 (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[22:30] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) Quit (Quit: Leaving.)
[22:31] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has left #ceph
[22:41] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) has joined #ceph
[22:43] * tryggvil_ (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[22:44] * tryggvil (~tryggvil@15-80-126-149.ftth.simafelagid.is) Quit (Ping timeout: 480 seconds)
[22:44] * tryggvil_ is now known as tryggvil
[23:01] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Ping timeout: 480 seconds)
[23:01] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[23:10] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[23:11] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[23:26] * lofejndif (~lsqavnbok@28IAAIO4Y.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:39] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) Quit (Quit: Leaving.)
[23:41] * tryggvil (~tryggvil@163-60-19-178.xdsl.simafelagid.is) has joined #ceph
[23:42] * Kioob (~kioob@luuna.daevel.fr) has joined #ceph
[23:45] * eccojoke (~ecco@h94n3-lk-d2.ias.bredband.telia.com) Quit (Quit: This computer has gone to sleep)
[23:49] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:49] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:51] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) has joined #ceph
[23:51] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) Quit ()

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.