#ceph IRC Log

Index

IRC Log for 2013-04-04

Timestamps are in GMT/BST.

[0:00] * drokita (~drokita@199.255.228.128) Quit (Quit: Leaving.)
[0:00] <gregaf> how were you converting between debian and ubuntu? that sounds like a process that could pretty easily knock stuff out of commission inadvertently
[0:01] <tchmnkyz-> i would take one osd out at a time
[0:01] <tchmnkyz-> wait for the cluster to fully rebuild with one less node
[0:01] <tchmnkyz-> i woudl start with doing a out for that node
[0:01] <tchmnkyz-> once return to full heath iw ould then remove it from the crush map
[0:01] <gregaf> ah, a host conversion, not a VM conversion, got it
[0:01] <tchmnkyz-> and remove it
[0:01] <tchmnkyz-> yea
[0:02] <tchmnkyz-> the vms are on proxmox frontend
[0:02] <gregaf> how many working VMs do you have against these two? (just trying to get an idea of if this is a probabilistic thing or something else)
[0:03] <gregaf> gotta tell you though, this doesn't sound at all familiar to me and I don't know how it would happen
[0:03] <gregaf> if the cluster thinks it's healthy your best bet is probably using disk recovery software on the disk image
[0:03] <tchmnkyz-> from what i can tell of 8 2
[0:03] <tchmnkyz-> maybe more only checked 4 so far
[0:04] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[0:05] * sagelap (~sage@67.sub-70-197-65.myvzw.com) has joined #ceph
[0:05] <sagelap> dmick: https://github.com/ceph/ceph/pull/196
[0:05] * sagelap (~sage@67.sub-70-197-65.myvzw.com) has left #ceph
[0:05] <tchmnkyz-> i have one vm that is read only that i am afraid to reboot
[0:07] <pioto> tchmnkyz-: well... maybe a good time to grab a copy of any important data from it...
[0:07] * sjustlaptop (~sam@206.29.182.186) has joined #ceph
[0:07] <tchmnkyz-> that is what i just did
[0:08] <pioto> for the one you exported... if you try something like... xxd /path/to/the/image |head
[0:08] <pioto> do you just see a bunch of 0's? or, just noise, or hwat?
[0:09] <tchmnkyz-> no it is not all 0's
[0:09] <tchmnkyz-> looks random
[0:10] <tchmnkyz-> http://pastebin.com/AiyANZ27
[0:10] <pioto> so... that makes me think it's something that was done in the vm itself
[0:11] <tchmnkyz-> so then no chance of fixing?
[0:11] <pioto> if, say, you lost one of the objects behind the rbd, i think you'd see all 0's
[0:11] <tchmnkyz-> k
[0:11] <pioto> i'm not at all an expert in ceph, though
[0:11] <pioto> so someone else may know better
[0:11] <pioto> but, well....
[0:11] <pioto> i think this may be a time to dig uyp your backups
[0:11] <pioto> and rebuild this vm
[0:14] <tchmnkyz-> this was a new setup we were starting
[0:14] <tchmnkyz-> i had not had time to back it up yet
[0:16] * ninkotech_ (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[0:17] * jjgalvez (~jjgalvez@12.248.40.138) Quit (Quit: Leaving.)
[0:18] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[0:19] * vata (~vata@2607:fad8:4:6:61f5:900a:247f:e5d1) Quit (Quit: Leaving.)
[0:19] <tchmnkyz-> i just tried tocreate a new partition table that was similar to the previous
[0:20] * madkiss (~madkiss@tmo-107-76.customers.d1-online.com) Quit (Quit: Leaving.)
[0:20] <dmick> tchmnkyz-: can you dump the first 512 bytes of the image into xxd?
[0:20] <tchmnkyz-> how would i do that
[0:20] <tchmnkyz-> never used xxd
[0:21] <dmick> the same way you did at 3:10
[0:21] <dmick> (03:10:15 PM) tchmnkyz-: http://pastebin.com/AiyANZ27
[0:21] <tchmnkyz-> k
[0:22] <tchmnkyz-> looking at the image it seems that more of the way down is all zeros
[0:23] <tchmnkyz-> so the first 10 line look like data
[0:23] <dmick> can you pastebin the first 512 bytes?
[0:24] * ninkotech (~duplo@ip-89-102-24-167.net.upcbroadband.cz) Quit (Ping timeout: 480 seconds)
[0:26] <tchmnkyz-> http://pastebin.com/Fm9q5FSv
[0:26] * The_Bishop (~bishop@2001:470:50b6:0:e8dc:72bc:e4bf:4533) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[0:28] <dmick> so that sorta looks like an MBR
[0:28] <tchmnkyz-> i mean it looks like there is data
[0:28] <tchmnkyz-> this was a rhel image
[0:29] <dmick> writing a program to decode this sensibly. stupid 16-bit-interpretation
[0:33] <tchmnkyz-> so far 3 vms brokeout of 8
[0:34] <dmick> this really doesn't look like the cluster losing data; it looks like something damaged the disk from inside the vm
[0:34] <tchmnkyz-> ok
[0:35] <tchmnkyz-> any possible way to recover it?
[0:36] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[0:37] <pioto> tchmnkyz-: backups? or maybe even just rbd snapshots if you have those (but then you'd lose any newer data)
[0:38] <pioto> or, pay some data recovery people
[0:38] * diegows (~diegows@190.190.2.126) has joined #ceph
[0:38] <pioto> or, well, just think of what you'd do if you had a physical disk that was corrupted in this sort of way
[0:41] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:41] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:44] <alram> tchmnkyz-: from the MBR it seems that the partition table is still there.
[0:45] <alram> was is 1st partition linux, 2nd parition swap and a 3rd one of type linux?
[0:47] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[0:48] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Read error: Connection reset by peer)
[0:49] * JohansGlock (~quassel@kantoor.transip.nl) has joined #ceph
[0:55] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[0:59] <dmick> tchmnkyz-: what alram said. Those partition entries look good, and one of them is marked active, even
[0:59] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[1:00] <tchmnkyz-> root@san03:~ # mount -t ext4 /dev/mapper/loop0p3 /tmp/
[1:00] <tchmnkyz-> mount: wrong fs type, bad option, bad superblock on /dev/mapper/loop0p3, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so
[1:00] <tchmnkyz-> that is what i get when i try to mount it
[1:03] <dmick> alram also points out that fdisk -l apparently does not work on a file; I don't know why
[1:03] <dmick> not sure what /dev/mapper/loop0p3 is
[1:04] <alram> tchmnkyz-: could you try to boot the VM with an ISO to get a rescue shell and try `fdisk -l`?
[1:05] <alram> if you still don't see a partition table gpart might help to recover it. I add some success with it.
[1:05] <alram> -add +had
[1:06] * jackhill (jackhill@pilot.trilug.org) has joined #ceph
[1:08] <tchmnkyz-> the vm where the primary disk as not affected
[1:08] <tchmnkyz-> the other disk in that system has no partition table
[1:09] <dmick> now I'm totally confused
[1:10] <tchmnkyz-> one vm lost the primary drive image
[1:10] <tchmnkyz-> another lost the secondary
[1:10] <dmick> were we looking at the lost primary image?
[1:10] * The_Bishop (~bishop@f052102191.adsl.alicedsl.de) has joined #ceph
[1:12] <dmick> well, I'll leave you to it then.
[1:15] * The_Bishop_ (~bishop@f052101215.adsl.alicedsl.de) has joined #ceph
[1:15] * sjustlaptop (~sam@206.29.182.186) Quit (Ping timeout: 480 seconds)
[1:16] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[1:18] <tchmnkyz-> sorry talking to boss
[1:18] <tchmnkyz-> the one i have theraw file of is a primary
[1:20] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[1:20] * The_Bishop (~bishop@f052102191.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[1:22] * The_Bishop__ (~bishop@f052098085.adsl.alicedsl.de) has joined #ceph
[1:22] <joelio> Just testing out the Ceph driver in 4.0 and I don't seem to be able to import an image into it from the marketplace, nor can I create a template that has it listed as a volative disk (doesn't seem to be an option)
[1:22] <joelio> am I missing something obvious?
[1:23] <dmick> joelio: huh?
[1:23] * The_Bishop_ (~bishop@f052101215.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[1:24] <dmick> what is 4.0, the marketplace, or template?
[1:25] * The_Bishop_ (~bishop@e177090078.adsl.alicedsl.de) has joined #ceph
[1:25] <joelio> doh, wrong channel
[1:25] * joelio should be inb #opennebula
[1:25] <joelio> apologies
[1:25] <dmick> whew
[1:25] <joelio> it's late!
[1:28] * The_Bishop (~bishop@e177088014.adsl.alicedsl.de) has joined #ceph
[1:28] <joelio> everything's runnign great on the backend.. it's the VM stuff I'm battling with now. Oh well, bed I guess. Will figure it out after sleep
[1:29] <dmick> vm configuration is often difficult it seems
[1:29] <dmick> the opportunities for middleware improvement are certainly there
[1:30] <tchmnkyz-> joelio: makes two of us
[1:31] * The_Bishop__ (~bishop@f052098085.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[1:31] <joelio> Ahh, think I've found the error anyway, tty issue in sudo - easy fix hopefully.
[1:31] <joelio> dmick: yes, definitely.. although it feels like it's moving in the right direction
[1:32] <joelio> half the battle is understanding the interactions between the layers
[1:32] * The_Bishop__ (~bishop@e177091075.adsl.alicedsl.de) has joined #ceph
[1:33] <tchmnkyz-> i was moving in the right direction then this happened and i feel like i am all teh way back at the begining
[1:33] * The_Bishop_ (~bishop@e177090078.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[1:35] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) Quit (Quit: gentleben)
[1:36] * The_Bishop__ (~bishop@e177091075.adsl.alicedsl.de) Quit (Read error: Operation timed out)
[1:36] * dpippenger (~riven@216.103.134.250) Quit (Remote host closed the connection)
[1:37] <dmick> tchmnkyz-: do you have some time to talk about this now?
[1:38] * The_Bishop (~bishop@e177088014.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[1:39] * The_Bishop__ (~bishop@f052102165.adsl.alicedsl.de) has joined #ceph
[1:44] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[1:44] * loicd (~loic@magenta.dachary.org) has joined #ceph
[1:44] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[1:48] * gentleben (~sseveranc@208.74.182.4.static.etheric.net) has joined #ceph
[1:49] * The_Bishop (~bishop@e177090094.adsl.alicedsl.de) has joined #ceph
[1:53] * The_Bishop_ (~bishop@e179000255.adsl.alicedsl.de) has joined #ceph
[1:53] * The_Bishop__ (~bishop@f052102165.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[1:57] * LeaChim (~LeaChim@02d9ee2d.bb.sky.com) Quit (Read error: Operation timed out)
[1:57] <tchmnkyz-> yea i do
[1:57] <tchmnkyz-> my boss just left
[1:57] <tchmnkyz-> basically told me to find a way to fix
[1:58] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[1:59] * The_Bishop (~bishop@e177090094.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[2:02] * The_Bishop__ (~bishop@e177088006.adsl.alicedsl.de) has joined #ceph
[2:02] * jjgalvez (~jjgalvez@12.248.40.138) Quit (Quit: Leaving.)
[2:04] * The_Bishop_ (~bishop@e179000255.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[2:14] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[2:27] <tchmnkyz-> dmick: i am here
[2:28] <dmick> hi.
[2:31] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[2:33] <tchmnkyz-> so any thoghts on correcting this?
[2:34] <dmick> let's focus on the VM that won't boot, claiming its disk is 'unbootable'. Was that disk image the one you extracted first 512 bytes from?
[2:34] <tchmnkyz-> yes
[2:35] <dmick> so that disk looks to me like it should be bootable, so I'm confused as to why it apparently is not working
[2:35] <tchmnkyz-> because when i try to mount it it has not superblock info
[2:35] <dmick> did you try alram's suggestion of booting an ISO into a rescue shell, and examining the disk with fdisk from that rescue shell?
[2:35] <tchmnkyz-> that was also after i tried to re-create the partition table
[2:35] <tchmnkyz-> let me make a fresh dump and give the first 512
[2:36] <dmick> so the original disk image is gone?
[2:36] <tchmnkyz-> no
[2:36] <tchmnkyz-> i dumped and tried to create a new partition table
[2:37] <dmick> I don't understand what that means; can you please be more specific in what you say you did
[2:37] <dmick> as in maybe "I ran this command"
[2:37] <tchmnkyz-> ok i did a dump from rbd, then took it into fdisk and created a new partition table
[2:37] <dmick> what does "did a dump from rbd" mean?
[2:38] <tchmnkyz-> rbd export image image.raw -p pool
[2:38] <dmick> there's a command
[2:38] <dmick> you did "rbd export" into a file.
[2:38] <tchmnkyz-> yes
[2:38] <dmick> and then you "took it into fdisk". What does that mean?
[2:38] <tchmnkyz-> then tried to take that file and run fdisk file
[2:40] <dmick> do you mean you issued the command "fdisk image.raw"?
[2:40] <tchmnkyz-> yea
[2:40] <tchmnkyz-> ok
[2:40] <tchmnkyz-> i did a fresh rbd export
[2:41] <tchmnkyz-> and got the first 512 of the image
[2:41] <tchmnkyz-> http://pastebin.com/tDWCPfiw
[2:41] <dmick> when I say "fdisk <file>", fdisk says to me "unable to read <file>: Inappropriate ioctl for device"
[2:42] <tchmnkyz-> really
[2:42] <tchmnkyz-> cuz mine works
[2:42] <tchmnkyz-> http://pastebin.com/3JP3rF8D
[2:43] <dmick> so you have some different fdisk. well, if it worked, then, did you see the three partitions that existed?
[2:43] <tchmnkyz-> no
[2:43] <tchmnkyz-> second pastebin is the output from it
[2:43] <dmick> the command you just pastebinned was fdisk -l <file>
[2:43] <dmick> that command is not fdisk <file>
[2:43] <dmick> it is fdisk -l <file>
[2:43] <tchmnkyz-> hang on
[2:43] <tchmnkyz-> will do the other too
[2:44] <tchmnkyz-> http://pastebin.com/nVUktHxN
[2:44] <tchmnkyz-> fixed one
[2:45] <dmick> I'm guessing fdisk is not doing the right thing on this file if it can't see the partition table
[2:45] <dmick> what release are you using?
[2:45] <dmick> (of Linux)
[2:45] <dmick> (distro, I should say)
[2:45] <tchmnkyz-> the vm was redhat el6
[2:45] <tchmnkyz-> teh box i am using for recovery is debian stable
[2:48] <dmick> trying to make any fdisk actually work against a file
[2:49] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[2:49] <dmick> amusingly, 'file' will show the partition table
[2:50] <dmick> what does that report on the original exported image?
[2:50] <tchmnkyz-> k
[2:50] <tchmnkyz-> that is the original exported image
[2:50] <tchmnkyz-> unmodified
[2:51] <dmick> I don't understand. I asked what 'file' reports on the original exported image. "that is the original exported image unmodified" doesn't look like file output.
[2:53] <dmick> (fdisk -l does work for me on an entire image; just not on the mbr alone. Apparently it does some partition validation, finds it wrong, and gives up without any error message)
[2:53] <dmick> tchmnkyz-: I'm asking you to run "file <imagename>" and tell me what it says.
[2:54] <dmick> tchmnkyz-: are you still there?
[2:56] <tchmnkyz-> yea
[2:56] <tchmnkyz-> i gave you it
[2:56] <dmick> no
[2:56] <tchmnkyz-> file image
[2:56] <tchmnkyz-> one sec
[2:56] <tchmnkyz-> sorr
[2:58] <tchmnkyz-> vm-5003-disk-1.raw: ACB archive data
[2:59] <dmick> that just makes no sense at all
[2:59] <dmick> is the VM an x86 VM?
[3:00] <tchmnkyz-> no 64bit
[3:00] <dmick> but Intel
[3:00] <tchmnkyz-> yes
[3:00] <dmick> not ARM or PowerPC or something
[3:00] <tchmnkyz-> ya Dual Xeon
[3:02] * KindOne (~KindOne@0001a7db.user.oftc.net) Quit (Remote host closed the connection)
[3:03] <dmick> well, I would still boot a VM of the same type from an ISO image and look at that disk with native utilities
[3:03] <dmick> I can't imagine why an image with the first 512 bytes you sent me would be so damaged
[3:04] * jjgalvez (~jjgalvez@12.248.40.138) Quit (Ping timeout: 480 seconds)
[3:09] <tchmnkyz-> it seems what happened is the parrent node maxed out resources the vm became deadlocked
[3:09] <tchmnkyz-> and the vm was killed
[3:09] <dmick> ok
[3:13] * Q (Q@2403-9000-2000-0200-0000-0000-0000-0020.ipv6.onqnetworks.net) has joined #ceph
[3:14] * Q is now known as Guest1141
[3:14] * Guest1141 is now known as Qu310
[3:14] * Qu310 is now known as Q310
[3:17] * KindOne (KindOne@0001a7db.user.oftc.net) has joined #ceph
[3:19] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Read error: Operation timed out)
[3:25] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[3:25] * loicd (~loic@magenta.dachary.org) has joined #ceph
[3:26] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[3:34] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[3:36] * gentleben_ (~sseveranc@208.74.182.4.static.etheric.net) has joined #ceph
[3:38] * gentleben_ (~sseveranc@208.74.182.4.static.etheric.net) Quit ()
[3:44] * gentleben (~sseveranc@208.74.182.4.static.etheric.net) Quit (Ping timeout: 480 seconds)
[4:22] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[4:27] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[4:34] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[4:35] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[4:37] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[5:06] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[5:17] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[5:26] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[5:38] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[5:56] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[6:02] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Quit: ChatZilla 0.9.90 [Firefox 19.0.2/20130307023931])
[6:03] * rekby (~Adium@ip-83-149-2-69.nwgsm.ru) has joined #ceph
[6:05] * rekby (~Adium@ip-83-149-2-69.nwgsm.ru) Quit ()
[6:06] * scuttlemonkey (~scuttlemo@fl-184-1-34-163.dhcp.embarqhsd.net) Quit (Ping timeout: 480 seconds)
[6:08] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[6:12] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[6:18] * jbarth (~jbarth@a.clients.kiwiirc.com) has joined #ceph
[6:36] * rekby (~Adium@2.92.10.160) has joined #ceph
[6:42] * rekby (~Adium@2.92.10.160) Quit (Quit: Leaving.)
[6:42] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[6:49] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[6:49] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) has joined #ceph
[6:55] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[7:01] * tchmnkyz- (~jeremy@ip23.67-202-99.static.steadfastdns.net) Quit (Ping timeout: 480 seconds)
[7:09] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[7:21] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[7:23] * jbarth (~jbarth@a.clients.kiwiirc.com) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[7:33] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[7:35] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) has joined #ceph
[7:50] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) Quit (Ping timeout: 480 seconds)
[7:53] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) has joined #ceph
[8:01] * norbi (~nonline@buerogw01.ispgateway.de) has joined #ceph
[8:10] * Hau_MI (~HauM1@login.univie.ac.at) has joined #ceph
[8:10] * HauM1 (~HauM1@login.univie.ac.at) Quit (Remote host closed the connection)
[8:11] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[8:18] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) has joined #ceph
[8:51] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) Quit (Quit: themgt)
[8:56] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[9:08] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: It's a dud! It's a dud! It's a du...)
[9:10] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) Quit (Quit: Leaving.)
[9:10] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:10] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has left #ceph
[9:20] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) has joined #ceph
[9:22] * loicd (~loic@185.10.252.15) has joined #ceph
[9:23] * leseb (~Adium@83.167.43.235) has joined #ceph
[9:25] * themgt (~themgt@97-95-235-55.dhcp.sffl.va.charter.com) has joined #ceph
[9:29] * Morg (b2f95a11@ircip3.mibbit.com) has joined #ceph
[9:34] * Hau_MI is now known as HauM1
[9:43] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[9:52] * themgt (~themgt@97-95-235-55.dhcp.sffl.va.charter.com) Quit (Quit: Pogoapp - http://www.pogoapp.com)
[9:54] * l0nk (~alex@83.167.43.235) has joined #ceph
[10:00] <norbi> anybody here who can help me? after upgrade vom 0.58 to 0.60 i have one crashing mds :(
[10:05] * sleinen (~Adium@2001:620:0:25:acbd:87e6:b7e9:7c3d) has joined #ceph
[10:07] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[10:08] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) has joined #ceph
[10:14] * tnt (~tnt@dhcp92.vr.in-berlin.de) has joined #ceph
[10:17] <alexxy[home]> norbi: i have same problem
[10:17] <alexxy[home]> did you tryed master?
[10:20] * sleinen (~Adium@2001:620:0:25:acbd:87e6:b7e9:7c3d) Quit (Quit: Leaving.)
[10:20] * sleinen (~Adium@130.59.94.143) has joined #ceph
[10:20] * LeaChim (~LeaChim@02d9ee2d.bb.sky.com) has joined #ceph
[10:21] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[10:23] * sleinen1 (~Adium@130.59.94.143) has joined #ceph
[10:25] * sleinen2 (~Adium@2001:620:0:25:d01c:aeae:c95a:fce3) has joined #ceph
[10:28] * sleinen (~Adium@130.59.94.143) Quit (Ping timeout: 480 seconds)
[10:31] * sleinen1 (~Adium@130.59.94.143) Quit (Ping timeout: 480 seconds)
[10:33] * BillK (~BillK@124-148-251-34.dyn.iinet.net.au) has joined #ceph
[10:42] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[10:45] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[10:47] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:53] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[10:53] * rekby (~Adium@2.92.10.160) has joined #ceph
[10:54] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[10:58] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) has joined #ceph
[11:03] <joao> alexxy[home], see mailing list for reply regarding the monitor crash
[11:03] <joao> just sent it
[11:03] <alexxy[home]> joao: i have mds crash
[11:04] <alexxy[home]> while mon works
[11:05] <alexxy[home]> joao: i tryed to downgrade to 0.59
[11:05] <alexxy[home]> but it has mon crashes
[11:05] <alexxy[home]> while on 0.60 i have mds crashed
[11:05] <norbi> hehe same here :)
[11:06] * jlogan1 (~Thunderbi@2600:c00:3010:1:64ea:852f:5756:f4bf) Quit (Quit: jlogan1)
[11:06] <norbi> one mds is working fine, one is crashing
[11:06] <norbi> joao - upgrade from 0.58 to 0.60 has resolved my crashing osds ;)
[11:06] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[11:07] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) has joined #ceph
[11:09] <joao> I see the mds bug
[11:09] <joao> been trying to find a filed ticket for it to no avail yet
[11:13] <joao> found a close match to that bug, but it was resolved 1 year ago: http://tracker.ceph.com/issues/1195
[11:13] <joao> norbi, alexxy[home], could one of you file a bug for this issue on the tracker?
[11:16] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[11:26] <alexxy[home]> joao: after applying mentioned patch to 0.59 mon and mds works
[11:26] <alexxy[home]> but mds now in status up:replay
[11:26] <alexxy[home]> and clients cant mount ceph fs
[11:27] <norbi> can do that, yust loolking for the "fill a bug" button :)
[11:27] <joao> alexxy[home], maybe this one? http://tracker.ceph.com/issues/4618
[11:29] <alexxy[home]> looks like
[11:29] <norbi> where is the button to fill a bug ?
[11:29] <alexxy[home]> but now i have osd crashed
[11:30] <alexxy[home]> sd.0 0 The disk uses features unsupported by the executable
[11:30] <alexxy[home]> *osd.0 0 The disk uses features unsupported by the executable
[11:30] <joao> norbi, select project, click on 'New issue'
[11:30] <joao> mds related issues should be filed under 'fs'
[11:30] <norbi> ah ok ! thx
[11:30] <joao> http://tracker.ceph.com/projects/cephfs
[11:33] <joao> I clearly need a new computer just to run thunderbird
[11:33] <joao> this thing is a cpu hog
[11:36] <alexxy[home]> joao: are there any incompatible changes in master? so i cannot rollback to 0.60 or 0.59?
[11:38] <norbi> realy ugly bugreport :)
[11:38] <norbi> http://tracker.ceph.com/issues/4644
[11:38] <joao> alexxy[home], you should be able to roll back the monitor to 0.59; no idea about the mds or the osd
[11:38] <joao> I believe you just cannot rollback the osd to 0.58 given disk format changes
[11:39] <joao> norbi, for future reference, enclose the dump in <pre> ... </pre>
[11:39] <joao> but I'll fix that for you
[11:40] <joao> done
[11:40] <alexxy[home]> ohh
[11:40] <alexxy[home]> seems like now i have crashed osd =\
[11:41] <joao> also, any logs you may have that may be useful to whoever looks into that, either point to them on the ticket or, if they have an appropriate size just attach them there
[11:42] <norbi> ok thx !
[11:45] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Read error: No route to host)
[11:46] * topro (~topro@host-62-245-142-50.customer.m-online.net) has joined #ceph
[11:56] <LeaChim> Hi, one of my monitors appears to have gone wrong. All 3 of the monitor processes are running, but the cluster state reports HEALTH_WARN, and states that one of the monitors is down.
[11:58] <LeaChim> It's currently writing a message to the log ~3 times a minute looking like: 2013-04-04 10:57:16.297816 7fd64b32f700 10 mon.hotcpc8912@2(peon) e3 ms_verify_authorizer 10.252.24.32:0/7453 client protocol 0
[12:00] <LeaChim> Any suggestions on how I work out what it thinks it's doing?
[12:05] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) Quit (Ping timeout: 480 seconds)
[12:06] <joao> LeaChim, 'ceph health detail' should give you insight on why it's HEALTH_WARN
[12:06] * rahmu (~rahmu@83.167.43.235) Quit (Remote host closed the connection)
[12:08] * bitserker (~bitserker@186.36.79.188.dynamic.jazztel.es) has joined #ceph
[12:09] <bitserker> hi
[12:10] <bitserker> this is the right place to ask about interaction with libvirt/ceph?
[12:11] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (Read error: Operation timed out)
[12:11] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[12:12] <matt_> bitserker, I have a fair bit of experience with ceph/libvirt. What's your problem?
[12:13] <bitserker> matt: hi matt. I'm trying to attach a volume using virsh but the command throws me this error
[12:13] <bitserker> matt: error: Failed to attach device from instance-00000104.xml
[12:13] <bitserker> error: operation failed: open disk image file failed
[12:14] <bitserker> matt: if i look in logs... debug : qemuMonitorIOWrite:474 : QEMU_MONITOR_IO_WRITE: mon=0x7f2f2800c470 buf={"execute":"human-monitor-command","arguments":{"command-line":"drive_add dummy file=rbd:volumes/volume-99ed8dd7-646e-4364-97a1-8cf5f11f5629:mon_host=nova-volume-lnx001\\:6789,if=none,id=drive-virtio-disk3,format=raw,cache=writeback"},"id":"libvirt-2227"}
[12:14] <bitserker> matt: look this -> mon_host=nova-volume-lnx001\\:6789
[12:14] <bitserker> matt: it's malformed?
[12:14] <matt_> so you're trying to live attach a drive?
[12:15] <bitserker> matt: yes, its a running machine
[12:15] <matt_> are you specifying the monitor from the command line or relying on the ceph.conf file?
[12:16] <bitserker> matt: i specify the monitor in xml
[12:16] <bitserker> matt: http://pastebin.com/mpbEGdLu
[12:17] <bitserker> matt: i have ceph and libvirt in separate hosts
[12:17] <bitserker> matt: i guess that libvirt dont read ceph.conf properly....
[12:17] <matt_> it's probably better you still have the ceph.conf file on the VM host also
[12:17] <matt_> that way you don't need to specify monitors. It will pick it up from the ceph.conf file
[12:18] <bitserker> matt: ok
[12:18] <bitserker> matt: im using openstack with the same problems
[12:18] <matt_> In my case I use puppet to push out a copy to each host which makes it nice and easy
[12:18] <bitserker> matt: cool
[12:19] <bitserker> matt: it seems openstack dont take the mon address of ceph.conf
[12:20] <alexxy[home]> joao: http://bpaste.net/show/88882/
[12:20] <bitserker> matt: but i have also a copy of ceph.conf in the host
[12:20] <alexxy[home]> log from crashed mds
[12:20] <matt_> bitserker, what version of libvirt are you using?
[12:21] <bitserker> matt: 0.9.13-0ubuntu12.2~cloud0
[12:22] <matt_> bitserker, this might be the bug you're experiencing http://www.mail-archive.com/libvir-list@redhat.com/msg66720.html
[12:22] <bitserker> matt: O_o
[12:23] <matt_> in which case an updated version of libvirt should fix it
[12:24] <bitserker> matt: ok i try it
[12:24] <bitserker> matt: thanks a lot!
[12:25] <joao> alexxy[home], could you please file a bug with that crash and attach the log to it?
[12:25] <matt_> bitserker, just to double check. Does nova-volume-lnx001 correctly resolve to an IP when you ping it?
[12:25] <joao> matt_, still have that monitor store around?
[12:25] <matt_> joao, the original 0.58 version or the 0.59 version that is crashing?
[12:26] <bitserker> matt: yes
[12:26] <joao> hopefully both, but the 0.59 would be fine as well
[12:26] <bitserker> matt: i also check the ports with telnet
[12:27] <bitserker> matt: all works fine
[12:27] <matt_> joao, I'm currently running 0.60 with the same issue
[12:27] <joao> matt_, yeah, it hasn't been fixed yet
[12:27] <alexxy[home]> joao: i cannot get reg mail
[12:27] <alexxy[home]> =)
[12:28] <joao> couldn't make it yesterday, had a lot going on and didn't manage to get to the fix in time
[12:28] <matt_> joao, did you just want me to tar it up and attach it to the bug report?
[12:28] <joao> matt_, I'll arrange a place for you to drop it
[12:30] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) has joined #ceph
[12:30] * goldfish (~goldfish@91.215.166.4) Quit (Ping timeout: 480 seconds)
[12:30] <joao> alexxy[home], weird, I'll file the bug for you and pass that to someone with admin permissions on the tracker
[12:32] <joao> alexxy[home], actually, looks like the bug norbi filed is the same
[12:32] <joao> http://tracker.ceph.com/issues/4644
[12:32] <joao> I'll attach your log to it
[12:34] <joao> done
[12:38] <norbi> hope you find the bug :)
[12:38] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[12:44] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[12:45] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[12:47] <bitserker> matt: there are any way to specify explicitly where is ceph.conf in openstack or libvirt? i have apparmor ok
[12:47] * loicd (~loic@185.10.252.15) Quit (Ping timeout: 480 seconds)
[12:47] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[12:48] <bitserker> matt: i think this is the problem.
[12:49] <matt_> I don't believe openstack support the ceph.conf file yet, I think it will be in the next release
[12:49] <matt_> libvirt defaults to /etc/ceph/ceph.conf and I'm not aware of any way of changing that
[12:49] <matt_> it may actually be a librbd setting but I'm not 100% on that, sorry
[12:50] * ivotron (~ivo@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[12:51] <bitserker> matt: ok np
[12:51] <bitserker> matt: tnx
[13:07] <nz_monkey_> I am really excited about the send/receive support being added
[13:07] <nz_monkey_> it is going to make backups super easy for us
[13:13] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[13:13] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[13:21] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[13:27] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) has joined #ceph
[13:30] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[13:32] <matt_> nz_monkey_, is send/receive between ceph clusters or something different?
[13:37] <dosaboy> is it normal that [mon.] does not appear in output of ceph auth list?
[13:37] <nz_monkey_> matt_: Different, it allows for dumping changes between snapshots to file or stdout, and also receiving them back in from file or stdin
[13:37] <dosaboy> or any mons for that matter
[13:37] <nz_monkey_> matt_: you in theory could use it to "replicate" between seperate clusters
[13:38] <matt_> nice
[13:38] <matt_> nz_monkey_, Do you happen to have a link with more info?
[13:38] <nz_monkey_> http://tracker.ceph.com/issues/4444
[13:39] <nz_monkey_> I vaguely remember some discussion on the dev list too
[13:40] <matt_> Thanks!
[13:46] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[13:47] <dosaboy> ok so if I specify -i <name> in CEPH_ARGS, that seems to override -n client.<name> on the command line. Is that expected?
[13:47] <dosaboy> would it not be better to allow command line to override if client.$id is specified
[13:49] * rekby (~Adium@2.92.10.160) Quit (Read error: Operation timed out)
[13:52] * goldfish (~goldfish@91.215.166.4) has joined #ceph
[13:57] <fghaas> dosaboy: apparently -i <id> beats -n <client>.<id>
[13:57] <fghaas> does the envar override even if you invoke ceph with -i?
[13:59] * dosaboy tries
[13:59] * The_Bishop_ (~bishop@e179015205.adsl.alicedsl.de) has joined #ceph
[14:02] <dosaboy> fghaas: yep, --id in CEPH_ARGS seems to overule any command ine ID option
[14:02] <dosaboy> which feels like a bug
[14:03] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[14:03] * The_Bishop__ (~bishop@e177088006.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[14:08] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[14:16] * The_Bishop__ (~bishop@e179012145.adsl.alicedsl.de) has joined #ceph
[14:18] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[14:18] * soren (~soren@hydrogen.linux2go.dk) has joined #ceph
[14:19] * The_Bishop_ (~bishop@e179015205.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[14:21] <soren> What's the easiest way to wipe a cephfs? I really just want to start from scratch, but without having to redo all the authentication stuff, etc.
[14:21] <infernix> # ceph osd stop osd.12
[14:21] <infernix> unknown command stop
[14:21] <infernix> does that seem right?
[14:22] <infernix> I'm following the troubleshooting guide but osd stop seems to not exist
[14:22] <infernix> did that change in bobtail and is the guide outdated?
[14:24] * scuttlemonkey (~scuttlemo@fl-184-1-34-163.dhcp.embarqhsd.net) has joined #ceph
[14:24] * ChanServ sets mode +o scuttlemonkey
[14:32] <soren> I'm using ceph under Ubuntu Raring. I've mounted the cephfs's root under /mnt. I have a previously created a snapshot (mkdir /mnt/.snap/a). Now I've added a dir under /mnt (mkdir /mnt/wtf) and in there, I'm trying to create a new snapshot (mkdir /mnt/wtf/.snap/a). This hangs.
[14:48] * goldfish (~goldfish@91.215.166.4) Quit (Ping timeout: 480 seconds)
[14:51] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[14:52] * goldfish (~goldfish@91.215.166.4) has joined #ceph
[14:54] * norbi (~nonline@buerogw01.ispgateway.de) Quit (Quit: Miranda IM! Smaller, Faster, Easier. http://miranda-im.org)
[14:55] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Remote host closed the connection)
[14:55] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[14:56] <stiller> I'm probably better off upgrading to 0.60 before trying PG split right?
[14:56] <infernix> initctl stop ceph-osd id=X
[14:56] * Morg (b2f95a11@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[14:56] <infernix> so that's how it's done.
[14:59] <soren> Hm. It works after a reboot. That's not exactly reassuring.
[15:08] * The_Bishop_ (~bishop@e179010094.adsl.alicedsl.de) has joined #ceph
[15:08] * The_Bishop__ (~bishop@e179012145.adsl.alicedsl.de) Quit (Read error: Connection reset by peer)
[15:13] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[15:25] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[15:26] * rahmu (~rahmu@83.167.43.235) Quit (Read error: Connection reset by peer)
[15:27] * madkiss (~madkiss@tmo-106-146.customers.d1-online.com) has joined #ceph
[15:33] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[15:53] <Azrael> scuttlemonkey: hey dude
[15:55] <scuttlemonkey> hey there
[15:57] <Azrael> scuttlemonkey: so it looks like ceph's chef cookbook works better for Debian Wheezy
[15:57] <Azrael> scuttlemonkey: still some debian-specific sections needed to be added
[15:58] <Azrael> scuttlemonkey: and there's also one little bug in the template for /etc/ceph/ceph.conf
[15:59] <Elbandi_> is any way to drop mds journal? so mds load the last stable state
[16:00] <Elbandi_> same as mount ext4 as ext2
[16:00] <scuttlemonkey> Azrael: glad it's not too terribly far afield
[16:04] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:09] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) has joined #ceph
[16:22] <Azrael> scuttlemonkey: do you know what the requirements and/or restrictions are for naming of OSD's in ceph.conf?
[16:23] <Azrael> e.g. *must* the names be [osd.#]?
[16:23] <Azrael> or can they be [osd.somehost.#]?
[16:23] <Azrael> please say [osd.somehost.#]... please say [osd.somehost.#]...
[16:23] <Azrael> and must the osd ids be contiguous? and must they start with 0?
[16:23] <Azrael> and why is the sky blue?
[16:24] <Azrael> and what is the answer to life the universe and everything?
[16:27] <fghaas> Azrael: as for contiguous, I now that they don't have to be, but a generated crushmap will fill in the blanks with dummy devices :)
[16:29] * rekby (~Adium@2.92.10.160) has joined #ceph
[16:29] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[16:29] <Azrael> haha
[16:29] <Azrael> sooo
[16:30] <Azrael> for config management (puppet, chef, etc)
[16:30] <Azrael> do you or anybody have a good method for automating ids?
[16:30] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Quit: Leaving.)
[16:38] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[16:54] * sjustlaptop (~sam@67-203-191-242.static-ip.telepacific.net) has joined #ceph
[16:55] <scuttlemonkey> Azrael: sorry, xchat didn't bonk me
[16:56] <scuttlemonkey> yeah, like florian said...incremental is best
[16:56] <scuttlemonkey> (or kinda what he said :P)
[16:56] <scuttlemonkey> as for automating ids you can see the puppet stuff here: https://github.com/enovance/puppet-ceph/blob/master/templates/ceph.conf-osd.erb
[16:57] <fghaas> well afaik the crushmap dummy devices don't really have any negative impact, other than making the crushmap a bit wordy and confusing (at least that's what I recall sage telling me in .au)
[16:58] <scuttlemonkey> yeah, crushmap gets messy in a hurry
[16:58] <scuttlemonkey> had someone who wanted to do osd.100, osd.200, etc
[16:58] <scuttlemonkey> or maybe .101, .201 or something, I forget
[16:59] <fghaas> scuttlemonkey: that's actually quite reasonable, considering that having overlapping osd ids is really bad
[16:59] <fghaas> so why not use 100s increments from host to host
[17:00] <scuttlemonkey> iirc it came when they went back and started trying to interject things in between
[17:00] <scuttlemonkey> although this was quite a while ago...there may have been a bug or somesuch that was giving them issue
[17:00] <scuttlemonkey> nfc what that would look like now
[17:01] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) Quit (Quit: gerard_dethier)
[17:02] <scuttlemonkey> just occurred to me I haven't actually strolled through the new ceph-deploy stuff to see how they are doing things
[17:02] <scuttlemonkey> I should do that
[17:03] <scuttlemonkey> fghaas: hey, have you played with any of the incremental snapshot stuff in rbd?
[17:04] <fghaas> scuttlemonkey: no sir
[17:04] <scuttlemonkey> hm, k
[17:04] <scuttlemonkey> writing up some of Josh D's thoughts and was looking for an outside perspective on use cases
[17:05] <fghaas> also, I need to talk to joshd in portland how to make nova+glance magically do the right thing for images, snapshots and volumes... that doesn't always quite work as advertised :)
[17:05] <fghaas> with rbd that is, obviously
[17:07] <topro> fghaas: having overlapping osd ids is bad in terms of resulting in technical problems or administrability?
[17:07] <scuttlemonkey> nova?
[17:08] <fghaas> scuttlemonkey: yeah, the way nova interacts with rbd via libvirt to attach volumes and/or boot from them
[17:08] <scuttlemonkey> ahh
[17:09] <fghaas> topro: I doubt it's allowed, i.e. if you're firing up osd.42 and there is already a osd.42 in the cluster, I would expect the monitor to just not hand out map leases to it
[17:09] <fghaas> (maybe someone could hit me with a clue x 4 if I'm misguided on this, thank you)
[17:10] <topro> fghaas: ah, ok. I though with overlapping you meant something like host1 with osd.1 osd.3 osd4 and host 2 with osd.2 osd.5 osd.6
[17:11] <fghaas> ah no, that's fine... if confusing
[17:11] <topro> ok, heart rate is coming down... slowly ;)
[17:11] * BMDan (~BMDan@static-108-44-155-130.clppva.fios.verizon.net) has joined #ceph
[17:12] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:16] * vata (~vata@2607:fad8:4:6:a8fa:c0b6:51d0:4934) has joined #ceph
[17:16] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[17:16] <Azrael> ok
[17:16] <Azrael> maybe i'm confused....
[17:17] <Azrael> every OSD in your cluster needs an entry in ceph.conf with [osd.#], correct?
[17:17] * diegows (~diegows@190.190.2.126) has joined #ceph
[17:18] <joelio> Anyone here using Ceph w/ OpenNebula?
[17:18] <scuttlemonkey> joelio: haven't really beat it up, but I was just getting ready to announce the work they did w/ ON 4 today
[17:19] <joelio> scuttlemonkey: well, I'm having quite a few issues with it
[17:19] <scuttlemonkey> yeah?
[17:19] <joelio> the drivers seem broken in Ubuntu
[17:19] <scuttlemonkey> hrm
[17:19] <BMDan> Azrael: In the vast majority of configurations, yes.
[17:20] <Azrael> and each # must be unique
[17:20] <Azrael> it seems that would become confusing with large numbers of osd's
[17:20] <Azrael> as to what osd runs on what server
[17:20] <BMDan> Though mine are very minimal; each is simply "[osd.1]", "host=storagenode1"; "[osd.2]", "host=storagenode2", etc.
[17:20] <scuttlemonkey> joelio: yer not using xen are you?
[17:21] <joelio> scuttlemonkey: stuff like inconsistent use of sudo so images get root ownership and can't be removed as it's trying to remove as oneadmin user (breaks image imports). I've fixed up that but there are still deployment troubles
[17:21] <BMDan> Azrael: Why do you care which OSDs are on which server?
[17:21] <joelio> scuttlemonkey: god, no
[17:21] <joelio> scuttlemonkey: libvirt kvm
[17:21] <scuttlemonkey> hehe, ok...someone last week was trying to do that, so I thought I'd ask :)
[17:21] <Azrael> BMDan: i'm planning on 14 disks per server, thus 14 independent OSD daemons
[17:22] <joelio> heh, no.. I'm au fait with the Ceph stuff now.. it's trying to reverse engineer several closely flying hacky scripts
[17:22] <joelio> on the ONE side
[17:22] <Azrael> BMDan: just nice to have that predictable
[17:22] <Azrael> BMDan: but even more so, i'm attempting to automate this with chef
[17:23] <joelio> scuttlemonkey: I'd definitely test it out before blogging it
[17:23] <joelio> (even if just for my sanity, haha!)_
[17:23] <Azrael> BMDan: and i'd like to not have to hand define osd id's for 60 servers with 14 disks
[17:23] * sjustlaptop (~sam@67-203-191-242.static-ip.telepacific.net) Quit (Ping timeout: 480 seconds)
[17:23] <scuttlemonkey> joelio: hehe, I stood up the basics and it seemed ok
[17:23] <scuttlemonkey> what flavor of ubuntu you on?
[17:23] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Ping timeout: 480 seconds)
[17:24] <joelio> scuttlemonkey: 12.04 LTS on the whole.. I bumped this to 12.10 for later libvirt but that's not the issue - other local datastores are fine
[17:24] <scuttlemonkey> k
[17:24] <Azrael> BMDan: but hmmmm wait a second. i'm starting to put together that its the osd UUID that really matters, not the ID in [osd.id] in ceph.conf
[17:26] <BMDan> Azrael: I did it via a script. for i in $(seq 0 47); do echo "[osd.$i]" ; echo " host = storagenode"$(($i/12))" ; done >> ceph.conf
[17:26] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[17:26] <BMDan> Divvies them up, 12 per storage node, between storage nodes 0 through 3, inclusive.
[17:26] <BMDan> Modify appropriately, ofc.
[17:26] <Azrael> yup
[17:26] <Azrael> ok
[17:26] <Azrael> nice
[17:27] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[17:27] <scuttlemonkey> joelio: ok, guess I'll set it back up from scratch to see what's up...maybe poke jmelis over at opennebula when he gets in
[17:27] <scuttlemonkey> thanks for letting me know
[17:27] <Azrael> automating the definitions is easy. i think i was just stressing out more than i needed to be, thinking the $i in [osd.$i] actually mattered somewhere in ceph land, besides just having to be unique in ceph.conf
[17:28] <BMDan> So, I asked this question a while back.
[17:28] <BMDan> I wanted to do x0yy, where "1022" would represent disk #22 on server #1.
[17:29] <absynth> you shouldn't.
[17:29] <BMDan> I was informed that keeping them sequential was best, and that gaps were ill-advised.
[17:29] <absynth> yep
[17:29] <BMDan> Right, I'm sharing it with Azrael.
[17:29] <BMDan> So they're not entirely simply unique identifiers.
[17:29] <Azrael> yeah
[17:29] <Azrael> so its important
[17:29] <Azrael> but the order doesn't matter
[17:29] <absynth> i am not sure you can even still give OSDnums when creating?
[17:29] <BMDan> Yeppers.
[17:30] <Azrael> thats why a nameing like [osd.$hostname.$id] would be great
[17:30] <BMDan> absynth: /me boggles a bit. Explain?
[17:30] <joelio> scuttlemonkey: I've found updated drivers on teh redmine tracker! I'll try those now (not included in 4.0 beta)
[17:30] <BMDan> Azrael: Well, putting hostname in there is dangerous; what happens when you consolidate two storage nodes onto a much bigger storage node down the line?
[17:30] * joelio crosses figngers
[17:30] <Azrael> yep
[17:31] <scuttlemonkey> joelio: ah hah! Let me know how that goes...would be important to mention if I spew blog-ish-ness :)
[17:31] <absynth> well, i am not sure if ceph osd create <num> still accepts the <num> or silently throws it away
[17:31] <absynth> but i havent done that in a while, so
[17:32] <Azrael> couldn't/shouldn't the mon nodes just assign each osd an id? and keep its list there, completely internal to ceph? rather than admins hand assigning in ceph.conf?
[17:32] <absynth> last time i checked, non-sequential IDs were frowned upon because, essentially, each out-of-sequence OSDnum was somehow converted into a "missing" osd or something
[17:32] <absynth> Azrael: that is definitely worthy of discussion
[17:32] <absynth> however, admins need to have a shorthand identifier
[17:32] <Azrael> true
[17:32] <absynth> because you do interact with the OSDs
[17:34] <janos> i'd like to divorce id's from names
[17:34] <janos> with tools that allow you to use either with appropriate flags
[17:34] <janos> let the system manage id's and people manage names
[17:35] <joelio> scuttlemonkey: yes, it supports v2 driver too by the looks.. promising!
[17:36] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Quit: Ex-Chat)
[17:36] <joelio> scuttlemonkey: http://dev.opennebula.org/issues/1796
[17:37] * madkiss (~madkiss@tmo-106-146.customers.d1-online.com) Quit (Quit: Leaving.)
[17:38] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[17:39] <scuttlemonkey> joelio: that bug report says cephx is broken?
[17:39] <scuttlemonkey> http://opennebula.org/documentation:rel4.0:ceph_ds
[17:39] <scuttlemonkey> just says the keyring just needs to be in a specific location
[17:42] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[17:42] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[17:48] * sjustlaptop (~sam@184.169.30.7) has joined #ceph
[17:49] <joelio> scuttlemonkey: I've not looked at the auth stuff yet tbh.. I have the admin keyring and ceph.conf on the hypervisor node, so should be ok (I assume??)
[17:50] <joelio> this is a test cluster mind..
[17:52] <joelio> the patch for the updated drivers is in that ticket - it's not been pulled into mainline/beta deb by the looks
[17:52] <joao> matt_, still around?
[17:52] <rekby> Is any step by step guide for manual setup cluster?
[17:52] <rekby> manual create key, init dirs, add osd and etc, instead of 5-minute guide with auto-setup cluster?
[17:55] <matt_> joao, half alseep but yep, still here
[17:57] <fghaas> rekby: read the mkcephfs source code
[17:58] <joao> matt_, did you by any chance kept your 0.58 stores?
[17:58] <rekby> thanks :) i''do it, i thought it binary file
[17:59] * aliguori (~anthony@32.97.110.51) has joined #ceph
[17:59] <matt_> joao, I believe I do. I have a post-upgrade copy that has all of the old monitor files in it
[18:00] <joao> okay, good
[18:00] <joao> this appears to have been caused by a bug during conversion
[18:00] <joao> contrary to my initial assumption
[18:01] <matt_> joao, would that mean it isn't repairable?
[18:01] <joao> I believe it should work out just fine, but have to talk to sage first
[18:02] <joao> I'll update the bug report as soon as I know something for sure
[18:03] <matt_> ah, excellent. Would you like the old mon copy uploaded like the last?
[18:03] <joao> if you don't mind, that would be great
[18:03] <matt_> no trouble at all. Thanks for the update
[18:03] <joao> thanks :)
[18:04] <joao> I'll be back shortly, grabbing a snack
[18:05] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[18:05] * dxd828 (~dxd828@195.191.107.205) has joined #ceph
[18:06] * sjustlaptop (~sam@184.169.30.7) Quit (Read error: Connection reset by peer)
[18:08] * BillK (~BillK@124-148-251-34.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[18:08] <matt_> joao, upload is all done
[18:11] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) Quit (Ping timeout: 480 seconds)
[18:15] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) Quit (Quit: Leaving.)
[18:19] * madkiss (~madkiss@tmo-106-206.customers.d1-online.com) has joined #ceph
[18:20] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[18:20] * BillK (~BillK@58-7-139-26.dyn.iinet.net.au) has joined #ceph
[18:23] * rekby (~Adium@2.92.10.160) Quit (Quit: Leaving.)
[18:30] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Quit: Leaving.)
[18:38] <joao> matt_, thanks!
[18:40] * sjust-phone (~sjust@m800536d0.tmodns.net) has joined #ceph
[18:41] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Remote host closed the connection)
[18:42] * sleinen2 (~Adium@2001:620:0:25:d01c:aeae:c95a:fce3) Quit (Quit: Leaving.)
[18:42] * sleinen (~Adium@130.59.94.143) has joined #ceph
[18:43] * madkiss1 (~madkiss@tmo-098-41.customers.d1-online.com) has joined #ceph
[18:47] * madkiss (~madkiss@tmo-106-206.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[18:48] * alram (~alram@38.122.20.226) has joined #ceph
[18:50] * sleinen (~Adium@130.59.94.143) Quit (Ping timeout: 480 seconds)
[18:52] * leseb (~Adium@83.167.43.235) Quit (Quit: Leaving.)
[18:52] * leseb (~Adium@83.167.43.235) has joined #ceph
[18:52] * leseb (~Adium@83.167.43.235) Quit ()
[19:02] * dxd828 (~dxd828@195.191.107.205) Quit (Quit: Textual IRC Client: www.textualapp.com)
[19:03] * sage (~sage@76.89.177.113) Quit (Ping timeout: 480 seconds)
[19:05] * JohansGlock (~quassel@kantoor.transip.nl) has joined #ceph
[19:07] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[19:11] * BillK (~BillK@58-7-139-26.dyn.iinet.net.au) Quit (Read error: Operation timed out)
[19:11] * madkiss1 (~madkiss@tmo-098-41.customers.d1-online.com) Quit (Quit: Leaving.)
[19:12] <Elbandi_> what sould i do if mds dump-journal doens work?
[19:13] * sjustlaptop (~sam@m800536d0.tmodns.net) has joined #ceph
[19:14] * sage (~sage@76.89.177.113) has joined #ceph
[19:22] <mjevans> Elbandi_: check ceph -s (or was it -S ?) for the status of your cluster?
[19:23] <mjevans> I suggest reading the manual to confirm my memory
[19:25] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[19:25] <Elbandi_> osdc/Objecter.cc: In function 'void Objecter::handle_osd_op_reply(MOSDOpReply*)' thread 7ffed55bf700 time 2013-04-04 19:25:23.861700
[19:25] <Elbandi_> osdc/Objecter.cc: 1321: FAILED assert(client_lock.is_locked())
[19:25] <Elbandi_> ceph -s is ok
[19:26] <mjevans> if it says HEALHT_OK then you should have quorum and at least one copy of all data (in theory)
[19:26] <Elbandi_> this is interesting, becasue journal-check is good
[19:27] <mjevans> Don't Panic, but we've now left the my depth on this issue.
[19:27] <Elbandi_> i mean, it's started to read the journal
[19:34] * sjustlaptop1 (~sam@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[19:38] <gregaf> Elbandi_: I'm not sure, but I think dump-journal is just broken :(
[19:38] <Elbandi_> ok
[19:38] <Elbandi_> :(
[19:39] <mjevans> It would be useful if you let us know exactly which version you were using
[19:39] <Elbandi_> i use bobtail
[19:39] <gregaf> yeah: http://tracker.ceph.com/issues/4105
[19:39] <Elbandi_> 0.56.4
[19:39] <mjevans> well less useful for me but more useful for others with different (more) data
[19:40] <gregaf> alexxy[home]: I updated http://tracker.ceph.com/issues/4644 with some requests; could you grab your mds0_sessionmap object and more debugging and upload it?
[19:41] * bitserker (~bitserker@186.36.79.188.dynamic.jazztel.es) Quit (Quit: Saliendo)
[19:41] * sjustlaptop (~sam@m800536d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[19:42] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[19:44] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[19:45] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[19:55] <alexxy[home]> gregaf: i can post it somewhere
[19:55] <alexxy[home]> but there is a problem i cannot login to bugzie
[19:56] <alexxy[home]> gregaf: also i use it as cephfs
[19:58] <alexxy[home]> gregaf: http://bpaste.net/show/88999/ mds.log
[19:59] <alexxy[home]> gregaf: http://bpaste.net/show/89000/ session.map
[20:01] <gregaf> thanks
[20:05] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[20:05] * dpippenger (~riven@216.103.134.250) has joined #ceph
[20:10] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[20:20] * ctrl (~ctrl@95-28-159-246.broadband.corbina.ru) has joined #ceph
[20:20] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[20:20] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[20:30] * ivotron (~ivo@eduroam-225-108.ucsc.edu) has joined #ceph
[20:30] * jskinner_ (~jskinner@69.170.148.179) has joined #ceph
[20:37] * jskinner (~jskinner@69.170.148.179) Quit (Ping timeout: 480 seconds)
[20:38] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Read error: Connection reset by peer)
[20:38] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:44] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Quit: Leaving.)
[20:44] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[20:47] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[20:49] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[20:50] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[20:51] * sleinen1 (~Adium@2001:620:0:25:c94:54fb:9221:b836) has joined #ceph
[20:52] * sjustlaptop1 (~sam@c-98-234-186-68.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[20:53] * rekby (~Adium@2.92.10.160) has joined #ceph
[20:57] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:03] * rekby (~Adium@2.92.10.160) Quit (Quit: Leaving.)
[21:05] * sleinen1 (~Adium@2001:620:0:25:c94:54fb:9221:b836) Quit (Quit: Leaving.)
[21:06] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[21:10] * BillK (~BillK@58-7-189-73.dyn.iinet.net.au) has joined #ceph
[21:11] * rekby (~Adium@2.92.10.160) has joined #ceph
[21:14] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:18] * mega (~chatzilla@84.244.1.200) has joined #ceph
[21:20] <mega> Hello, need some help with keeping mons in quorum after upgrade 0.58->0.59->0.60 . Mons die constantly and I cannot make them last.
[21:20] * rekby (~Adium@2.92.10.160) Quit (Quit: Leaving.)
[21:22] <mega> Correction: mons do not die but rather stop responding. ceph restart mon get it going for a minute and then they die again.
[21:24] * mega is now known as mega_au
[21:24] <mega_au> Hello, need some help with keeping mons in quorum after upgrade 0.58->0.59->0.60 . Mons die constantly and I cannot make them last.
[21:24] <mega_au> Correction: mons do not die but rather stop responding. ceph restart mon get it going for a minute and then they die again.
[21:24] <BMDan> Sounds like they aren't heartbeating correctly. Is there a firewall between them (e.g. iptables) blocking their heartbeats?
[21:24] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[21:25] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[21:25] * jskinner_ (~jskinner@69.170.148.179) Quit (Read error: Connection reset by peer)
[21:26] <mega_au> Nothing... Just gigabit switch...
[21:27] <mega_au> And no iptables of course...
[21:29] <mega_au> During upgrade two mons have failed to upgrade from 0.58 to 0.59 . Copied mon stores away for safety and recreated mons completely - it started to sync but fails to complete.
[21:31] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[21:32] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:34] <BMDan> If there's a newer version of the monmap that the other mons don't have, then they'll drop out of quorum.
[21:34] <BMDan> Check whether the FS permissions are correct for where the mons are trying to write their monmaps.
[21:35] <gregaf> you should go look at the quorum status on the monitor that upgraded successfully — it's probably looking for the two that you nuked, and since it can't find them nothing is making any progress
[21:35] <mega_au> monmaps are the same, permissions are OK
[21:37] <mega_au> monmap e19: 5 mons at {0=172.16.64.200:6789/0,1=172.16.64.201:6789/0,2=172.16.64.202:6789/0,3=172.16.64.203:6789/0,4=172.16.64.204:6789/0}, election epoch 1812, quorum 0,3,4 0,3,4
[21:37] <mega_au> Currently everything 0.60
[21:37] <BMDan> Do you actually have five working mons, or just 3?
[21:38] <mega_au> So I have quorum with new mons but I lose it all the time and need to restart mons to keep them going.
[21:38] <mega_au> Three out five
[21:39] <mega_au> It rolls for a minute (may be 30 seconds) and then ceph -w shows: 2013-04-05 06:06:59.383792 7fa22f2cb700 0 -- 172.16.64.202:0/26777 >> 172.16.64.201:6789/0 pipe(0x7fa2200156f0 sd=4 :0 s=1 pgs=0 cs=0 l=1).fault
[21:40] <mega_au> And even when ceph -w stops I still have RBD images running OK.
[21:40] <gregaf> okay, so that's some kind of problem with the monitors losing communication repeatedly
[21:41] <gregaf> but you should remove the two non-existent monitors, because the reason it's causing you issues is that right now if you lose any of your monitors none of them can make progress (it needs 3 of 5, and you only have 3), when you are supposed to be able to lose one (which you can when you have 3 of 3)
[21:41] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) has joined #ceph
[21:42] <mega_au> But there no change of environment from 0.58 to 0.59 and so I guess it rules out network issues (unless there something I have missed in terms 0.59 requirements)
[21:43] <mega_au> I was expecting if I will get current keyring and monmap I can do ceph-mon mkfs which should get mon working again. Is it correct?
[21:44] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[21:44] <gregaf> ….no, not if you want any of your existing data
[21:45] <gregaf> remove the non-existent mons from the map, and then if you still have trouble joao or somebody will be able to help you look at taht
[21:45] <mega_au> Well... Of course I want to keep my data :)
[21:45] * sleinen (~Adium@2001:620:0:26:d0bf:fbcb:bba8:bcb3) has joined #ceph
[21:47] <BMDan> mega_au: Less of an "of course" than you'd think. I wiped my data several times a day for a few weeks straight while setting up Ceph. :)
[21:48] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[21:48] <mega_au> Well... I had my bad day blowing away everything already but for this purpose I back up whole RBD images daily... :)
[21:51] <mega_au> removed mon.2 at 172.16.64.202:6789/0, there are now 3 monitors
[21:54] <gregaf> Elbandi_: posted a journaler debugging patch which might provide more useful info; this issue is looking stranger all the time and doesn't seem to have anything to do with your multiple MDS setup so I'm quite intrigued
[21:55] * jlogan1 (~Thunderbi@2600:c00:3010:1:64ea:852f:5756:f4bf) has joined #ceph
[21:57] <mega_au> gregaf: having only 3 mons now but still no banana: 2013-04-05 06:26:51.073017 7f5472fb1700 0 monclient: hunting for new mon
[21:58] * ivotron (~ivo@eduroam-225-108.ucsc.edu) Quit (Ping timeout: 480 seconds)
[21:59] <gregaf> what are your mons saying when you look at their logs? are they moving in and out of quorum?
[21:59] <Elbandi_> gregaf: ok, i try it
[22:00] * Cube (~Cube@12.248.40.138) has joined #ceph
[22:00] * tnt (~tnt@dhcp92.vr.in-berlin.de) Quit (Ping timeout: 480 seconds)
[22:01] <Elbandi_> i have examined the log, and 201.00003f30 is the last chunk which has been queried
[22:01] * ivotron (~ivo@eduroam-225-108.ucsc.edu) has joined #ceph
[22:02] <mega_au> gregaf: nothing about quorum. It goes normally: 2013-04-05 06:30:31.109032 7ff0127fc700 0 log [INF] : pgmap v21232636: 576 pgs: 576 active+clean; 2061 GB data, 6191 GB used, 1868 GB / 8060 GB avail; 15937KB/s wr, 36op/s
[22:02] <mega_au> 2013-04-05 06:30:31.117802 7ff0127fc700 1 mon.3@1(leader).log v18170613 check_sub sending message to client.? 172.16.64.200:0/7713 with 2 entries (version 18170613)
[22:02] <mega_au> 2013-04-05 06:30:31.118320 7ff0127fc700 1 mon.3@1(leader).log v18170613 check_sub sending message to client.? 172.16.64.200:0/4919 with 2 entries (version 18170613)
[22:02] <mega_au> and then it stops with: 2013-04-05 06:31:11.049708 7fefc6fef700 0 -- 172.16.64.203:6789/0 >> :/0 pipe(0x7ff00c000f60 sd=30 :6789 s=0 pgs=0 cs=0 l=0).accept failed to getpeername 107 Transport endpoint is not connected
[22:02] <mega_au> 2013-04-05 06:31:19.945509 7fefc65e5700 0 -- 172.16.64.203:6789/0 >> 172.16.64.10:0/2963819212 pipe(0x7ff00c008350 sd=35 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 172.16.64.10:0/2963819212 (socket is 172.16.64.10:33173/0)
[22:04] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[22:05] <mega_au> 172.16.64.10 runs 0.55 client - can it cause some issue when connecting to 0.60 mon?
[22:06] * sjust-phone (~sjust@m800536d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[22:10] <mega_au> Disconnected 0.55 client - mons still die
[22:14] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) has joined #ceph
[22:17] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[22:20] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[22:22] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:23] * loicd1 (~loic@magenta.dachary.org) has joined #ceph
[22:23] * loicd (~loic@magenta.dachary.org) Quit (Read error: Connection reset by peer)
[22:25] * tnt (~tnt@pd95cfcb4.dip0.t-ipconnect.de) has joined #ceph
[22:27] <gregaf> mega_au: ah, you're running a mixed version system? It ought to work but those messages you're seeing indicate trouble at the protocol or transport layer
[22:28] <mega_au> I have six boxes running 0.60 only as ceph cluster, just a separate client running 0.55 (disabled it at the moment)
[22:28] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[22:28] * ctrl (~ctrl@95-28-159-246.broadband.corbina.ru) Quit ()
[22:29] <gregaf> I'd check into your network and ethernet card stability based on that output — ENOTCONN isn't an error code we generate
[22:32] * BillK (~BillK@58-7-189-73.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[22:35] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[22:43] <mega_au> Reading through logs I see this happening everytime mon stops: 7fb4a58d2700 5 mon.0@0(leader).paxos(paxos updating c 39766344..39766378) accept timeout, calling fresh election
[22:44] <gregaf> okay, so that would be a quorum change, like I was asking about before
[22:45] <gregaf> the "accept timeout" indicates that either there's a problem communicating with one (or more) of the monitors, or that one (or more) of them is taking a really long time to write data to disk
[22:45] * t0rn (~ssullivan@2607:fad0:32:a02:d227:88ff:fe02:9896) Quit (Quit: Leaving.)
[22:46] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) has joined #ceph
[22:46] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) Quit (Quit: Leaving.)
[22:48] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[22:51] <mjevans> That would indicate either too much disk data, insufficient link bandwidth, time desync, or that you might need to relax your thresholds at least in order to get stale nodes to resync.
[22:53] <mega_au> Disk space is OK, bandwith is OK, time is within limits (50 ms or less). How to relax thresholds? I guess mons just out of sync...
[22:54] <gregaf> well, you should check the logs on the monitors around the time of the failed proposal and see which of them didn't reply to the proposal
[22:54] <gregaf> and then try to figure out why
[22:55] <Elbandi_> gregaf: i uploaded the new log
[22:56] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[22:56] <gregaf> awesome, thanks
[22:56] <Elbandi_> there are sima line like (write_pos change read_pos change), thats my debug lines
[22:56] <Elbandi_> but no luck :(
[22:56] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) has joined #ceph
[23:05] * LeaChim (~LeaChim@02d9ee2d.bb.sky.com) Quit (Ping timeout: 480 seconds)
[23:05] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[23:06] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:06] * sleinen (~Adium@2001:620:0:26:d0bf:fbcb:bba8:bcb3) Quit (Quit: Leaving.)
[23:10] * loicd1 (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:11] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:12] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Ping timeout: 480 seconds)
[23:13] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[23:14] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) Quit (Quit: Leaving.)
[23:14] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:15] * LeaChim (~LeaChim@027f1746.bb.sky.com) has joined #ceph
[23:18] <BMDan> Quick question: is it acceptable (regardless of whether it's ideal) to have non-Ceph data in an OSD data directory?
[23:18] <gregaf> Elbandi_: not certain, but I think I see a bug in the journal fetching code — try adding "journaler prefetch periods = 20" to your MDS config and see if it gets any farther with that?
[23:21] <gregaf> in the top-level directory, or in the PG directories?
[23:22] <gregaf> the top-level directory won't break anything; the PG directories will
[23:22] <gregaf> BMDan: ^
[23:23] <BMDan> Ah, thanks. :)
[23:23] <BMDan> Makes sense.
[23:31] <mega_au> Hmm... Reading logs... It appears mon0 makes proposal and despatches it only to mon3 and does not call mon4 at all (I currently have only mon0, mon3 and mon4). Not sure why it does not contact mon4
[23:32] <gregaf> was mon4 in the quorum at that time? if it had previously failed to ack the others might have joined together without it
[23:36] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[23:38] <mega_au> No, it was not... So mon0 and mon3 joined together and forgot about mon4
[23:39] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:40] <mega_au> BTW: how to increase thresholds to allow mons to sync? I think it worth to try...
[23:40] * BMDan (~BMDan@static-108-44-155-130.clppva.fios.verizon.net) Quit (Quit: Leaving.)
[23:47] <Elbandi_> gregaf: 8G log, and still working... :>
[23:47] * vata (~vata@2607:fad8:4:6:a8fa:c0b6:51d0:4934) Quit (Quit: Leaving.)
[23:52] <Elbandi_> gregaf: i run with --journal-check
[23:52] <Elbandi_> and i think it's ok now
[23:53] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[23:54] <Elbandi_> mds.1.274 request_state up:resolve
[23:55] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[23:56] <gregaf> Elbandi_: sweet!
[23:58] <Elbandi_> need this a new patch?
[23:58] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit ()
[23:58] <gregaf> well, there's a bug in calculating how much of the journal it needs to read
[23:58] <gregaf> so yes, that will need patches to fix
[23:58] <gregaf> what I had you do is just a workaround for it
[23:58] <gregaf> you had a *very* large event in there of ~43MB
[23:59] <Elbandi_> maybe,
[23:59] <Elbandi_> i started to rsync ~100k files
[23:59] <gregaf> a very large directory, probably?

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.