#ceph IRC Log


IRC Log for 2012-06-21

Timestamps are in GMT/BST.

[0:12] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) has joined #ceph
[0:13] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[0:31] <tv_> have i told you guys how much i hate the ISC?
[0:31] <sagewk> nhm, dmick: it was a single machine. not sure why it broke.. doing the apt-get install manually worked fine, and now it's happy
[0:32] <tv_> trying to configure dhcpd the right way.. first 2000 lines of documentation are unhelpful
[0:34] * gohko_ (~gohko@natter.interq.or.jp) Quit (Read error: Connection reset by peer)
[0:34] * goedi (goedi@ Quit (Read error: Connection reset by peer)
[0:34] * goedi (goedi@ has joined #ceph
[0:35] * gohko (~gohko@natter.interq.or.jp) has joined #ceph
[0:36] * BManojlovic (~steki@ Quit (Remote host closed the connection)
[0:40] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) Quit (Quit: Zzzzzz)
[0:41] <dmick> if it helps, the Sun DHCP server was not better
[0:41] <dmick> and significantly less flexible
[0:41] <dmick> actually maybe that is better in and of itself
[0:42] <joshd> gregaf: added some comments to https://github.com/ceph/ceph/commit/3c05629691deb800e3c6e62e81f444a748e8857c
[0:43] <gregaf> joshd: yeah, I saw them coming in, thanks
[0:43] <joshd> cool, wasn't sure if your github notifications were working again
[0:46] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) has joined #ceph
[0:49] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) Quit ()
[0:57] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[0:57] <dmick> do we have a drop spot for customers to send us logs and/or coredumps?
[0:58] <dmick> (fully realizing this is something that needs to be controlled/checked for DoS)
[1:03] <sagewk> dmick: yeah
[1:10] * Ryan_Lane (~Adium@dslb-178-008-175-159.pools.arcor-ip.net) has joined #ceph
[1:16] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[1:59] * tv_ (~tv@aon.hq.newdream.net) Quit (Quit: tv_)
[2:03] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[2:14] * yehudasa_ (~yehudasa@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[2:58] * lofejndif (~lsqavnbok@9YYAAHCV1.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[3:14] * mosu001 (~mosu001@en-439-0331-001.esc.auckland.ac.nz) has joined #ceph
[3:15] <mosu001> Hi everyone, I had a problem with concurrent iozone access to a ceph mount point, so I decided it was time to update to ceph 0.47
[3:15] <mosu001> I also upgraded from openSUSE 11.3 to 12.1
[3:15] <mosu001> I then had to mkfs.btrfs, mount, ceph-osd, etc to get my OSDs back
[3:16] <mosu001> All the OSDSs now start ok, but all my pgs are stuck as stale and I'm not sure how to unstick them?
[3:16] <mosu001> Any ideas?
[3:17] <mosu001> ceph -s gives
[3:17] <mosu001> lp0:~/ogg # ceph -s
[3:17] <mosu001> health HEALTH_WARN 2304 pgs stale; 2304 pgs stuck stale
[3:17] <mosu001> monmap e1: 2 mons at {0=,1=}, election epoch 2, quorum 0,1 0,1
[3:17] <mosu001> osdmap e90: 12 osds: 12 up, 12 in
[3:17] <mosu001> pgmap v126648: 2304 pgs: 2304 stale+active+clean; 26614 MB data, 24018 MB used, 21416 GB / 22340 GB avail
[3:17] <mosu001> mdsmap e49: 1/1/1 up {0=1=up:replay}, 1 up:standby
[3:18] <mosu001> I've tried stooping and starting the system several times, but no luck
[3:27] <joshd> if you reformatted all your osd's filesystems, you'll need to re-initialize the cluster (i.e. mkcephfs)
[3:27] <ajm-> interesting, i missed that part reading that
[3:30] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[3:32] <mosu001> Will mkcephfs destroy the data on the disks?
[3:33] <ajm-> yes
[3:33] <ajm-> but you already did by mkfs.btrfs
[3:33] <The_Bishop> morpheus: with formatting ALL the OSDs you already delteted all content
[3:33] <ajm-> did you have important data on there?
[3:33] <The_Bishop> sry, i ment mosu001
[3:33] <mosu001> I did the mkfs.btrfs, mount, ceph-osd on a different test system and it is now up and running with the data intact (as far as I can tell)
[3:33] <ajm-> if you ran mkfs.btrfs, you erase the data, it creates a new filesystem
[3:33] <ajm-> overwriting the old one
[3:33] <ajm-> and all the contents
[3:34] <mosu001> So the data must have been restored from the OSD that was still working?
[3:34] <The_Bishop> it this is only on one OSD, the data survives, but not if you format everything
[3:34] <ajm-> to be clear, your doing mkfs.btrfs on the osd data partition, correct?
[3:34] <mosu001> OK, the steps I did were as follows:
[3:34] <mosu001> mkfs.btrfs /dev/sdb
[3:35] <mosu001> mount /dev/sdb /data/osd.11
[3:35] <ajm-> and you did that on ALL nodes at the same time?
[3:35] <mosu001> ceph-osd -c /etc/ceph/ceph.conf --mkfs --monmap /tmp/monmap
[3:36] <mosu001> yes, beacuse they were all failing to read the superblock
[3:36] <ajm-> the mkfs.btrfs step removed all the existing data
[3:36] <mosu001> Note, this is experimental, so the data is not critical
[3:37] <mosu001> OK, so in my other cluster which has 3 OSDs - 2 on one machine and 1 on the other with equal distribution among the 3 OSDs
[3:37] <mosu001> I d di the same steps on OSDS 0 and 1 and the data seems to be preserved
[3:37] <mosu001> Is this lucky or are the files likely to be corrupted
[3:37] <ajm-> i find it very unlikely
[3:38] <mosu001> The replication level was set to 2 as far as I know
[3:38] <ajm-> if you erased them both at the same time
[3:38] <ajm-> the data would all be gone
[3:38] <ajm-> its also very good to know it wasn't critical data as recovery would be non-trivial
[3:38] <mosu001> That's what I thought would happen, but it seems to be in the mount point...
[3:38] <ajm-> can you read it?
[3:38] <The_Bishop> yeah, with replication level 2 you should not remove/reformat more than 50% of the storage
[3:39] <mosu001> hold on I haven't tried yet...
[3:39] <ajm-> though if the mds'es start and you can mount the filesystem, you didn't erase things
[3:39] <ajm-> if you had it mounted it could just be a stale mount
[3:39] <mosu001> Yeah, the MDS and mount seem ok
[3:41] <mosu001> The data in the smaller system seems ok
[3:41] <ajm-> mosu001: it would likely be very difficult to try to figure things out in either event, but I can say you shouldn't do that if you want to keep your data :)
[3:41] <mosu001> Id did all the OSD (mkfs, mount, ceph-osd) with ceph running
[3:41] <ajm-> perhaps mkfs.btrfs failed on the OSD's on the smaller system
[3:41] <ajm-> but worked on the larger
[3:41] <mosu001> Yes, I realised the risk and was suprised when it seems ok on the smaller system
[3:42] <mosu001> OK, I thought perhaps it just fixed up the superblock and kept the data around
[3:42] <ajm-> are you 100% sure that mkfs.btrfs worked on the smaller system
[3:42] <ajm-> and didn't barf and say "I already see a filesystem here" ?
[3:42] <mosu001> I'll try mkcephfs on the larger system, I have a copy of this data elsewhere anyway
[3:43] <mosu001> I don't think so, I was watching pretty carefully
[3:43] <ajm-> ok, in that case I can't really explain it :)
[3:43] <mosu001> But it is definitely possible
[3:43] <ajm-> if you still have your terminals open check, its the easiest explanation
[3:43] <mosu001> So it was just re-mounting and running ceph-osd that fixed the superblock problem?
[3:44] <mosu001> Sorry the ssh session was finished
[3:44] <mosu001> Can I check the logs?
[3:45] <ajm-> no
[3:46] <mosu001> OK, I'll mkcephfs on the bigger system and see how that goes. Thanks for the advice
[3:46] <mosu001> Can I ask about the test I was doing?
[3:46] <ajm-> you can ask :)
[3:47] <mosu001> Thanks
[3:47] <mosu001> I had 5 ceph clients that had mounted the ceph FS
[3:47] <mosu001> The ceph FS has two MON/MDS nodes and two OSD nodes with 6 OSDs on each
[3:48] <mosu001> CRUSh replicated across the two OSD nodes (essentially a mirror)
[3:49] <mosu001> From each of the ceph clients I started an iozone test to a different file in the root ceph mount point
[3:49] <mosu001> It seemed like ceph hung...
[3:49] <mosu001> Note, I am not really a file system person!
[3:49] <mosu001> I was told that this might be due to locking on the inode for the ceph root directory...?
[3:50] <mosu001> I saw there were some syncing issues so I upgraded ceph (and haven't tried the test on the new system yet)
[3:50] <ajm-> hrm, performance benchmarks i am not the person to ask :)
[3:50] <ajm-> i honestly don't even have a ceph system atm
[3:51] <ajm-> are you using kernel client or fuse/
[3:51] <mosu001> kernel client
[3:57] <mosu001> mkcephfs seems to have done the trick, the pgs are peering now
[3:57] <mosu001> Looks a little like the data might still be there (FS is not empty...!)
[3:59] <mosu001> I'll see in a while and try to test initially to different directories first
[4:01] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) has joined #ceph
[4:01] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[4:03] * dmick is now known as dmick_away
[4:22] * Ryan_Lane1 (~Adium@dslb-094-223-088-003.pools.arcor-ip.net) has joined #ceph
[4:28] * Ryan_Lane (~Adium@dslb-178-008-175-159.pools.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[4:30] * Ryan_Lane1 (~Adium@dslb-094-223-088-003.pools.arcor-ip.net) Quit (Ping timeout: 480 seconds)
[4:37] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) Quit (Quit: Zzzzzz)
[5:01] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) has joined #ceph
[5:14] <sage> if anyone has machines locked they dont need, please unlock them!
[5:24] * nyeates (~nyeates@pool-173-59-239-237.bltmmd.fios.verizon.net) Quit (Quit: nyeates)
[5:30] <elder> I've got 3, and I need them all.
[5:31] <elder> Although I've got that run from this morning, it might have some locked, I haven't checked on it.
[5:54] <sage> almost done. lots of failures from the stupid mencoder+chef snafu
[6:04] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[6:08] <mosu001> See earlier posts for details, but I have stuck peering with the following output from ceph -w
[6:08] <mosu001> 2012-06-21 16:08:00.663366 osd.9 [WRN] slow request 4495.911644 seconds old, received at 2012-06-21 14:53:04.751669: osd_op(mds.0.5:21 100.00000000 [setxattr path (5),setxattr parent (13),tmapput 0~4089] 1.c5265ab3 RETRY) v4 currently delayed
[6:08] <mosu001> I checked the disks (connected via a RAID controller) and they seem fine, so not sure if I just need to wait or ...?
[6:09] <sage> 'ceph health detail' will tell you the problematic pgs, then 'ceph pg <pgid> query' will tell you why
[6:09] <sage> http://ceph.com/docs/master/ops/manage/failures/ has lots of good info
[6:09] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[6:09] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit ()
[6:10] <mosu001> lp0:~/ogg # ceph health
[6:10] <mosu001> HEALTH_WARN 447 pgs peering; 447 pgs stuck inactive; 447 pgs stuck unclean
[6:10] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[6:10] <mosu001> So there are a lot of pgs in ceph health detail
[6:11] <mosu001> I've got one of their ids and have done a ceph health query, but not sure what to look for?
[6:11] <mosu001> Follow the URL you posted?
[6:11] <sage> pastebin the whole thing?
[6:11] <sage> (the pg query result)
[6:12] <mosu001> ok, working on it
[6:14] <mosu001> Its at http://pastebin.com/ab8q7vVB
[6:19] <sage> hmm. not obvious why ti' snot making progress. try another random pg?
[6:19] <sage> you can also try bouncing an osd (ceph osd down 9) and see if that helps.. there are some rare races in older code that can cause it to get stuck like this
[6:21] <mosu001> Sorry sage, that one was peering but not stuck (according to ceph health detail)
[6:21] <sage> ah
[6:22] <mosu001> When I check a peering stuck one, e.g., ceph pg 1.c query it hangs...
[6:23] <sage> ceph pg map <pgid> will tell you which osd it's on
[6:23] <mosu001> HEALTH_WARN 447 pgs peering; 447 pgs stuck inactive; 447 pgs stuck unclean
[6:23] <mosu001> pg 1.13 is stuck peering, last acting [9,2]
[6:23] <mosu001> pg 2.12 is stuck peering, last acting [9,2]
[6:23] <mosu001> pg 1.c is stuck peering, last acting [0,8]
[6:23] <mosu001> pg 0.d is stuck peering, last acting [0,8]
[6:23] <sage> that osd is probably down
[6:23] <mosu001> pg 1.8 is stuck peering, last acting [9,1]
[6:23] <mosu001> pg 0.9 is stuck peering, last acting [9,1]
[6:23] <mosu001> pg 2.b is stuck peering, last acting [0,8]
[6:23] <mosu001> pg 2.5 is stuck peering, last acting [1,9]
[6:23] <mosu001> pg 1.6 is stuck peering, last acting [1,9]
[6:23] <mosu001> pg 0.7 is stuck peering, last acting [1,9]
[6:23] <mosu001> pg 2.7 is stuck peering, last acting [9,1]
[6:23] <mosu001> pg 2.0 is stuck peering, last acting [9,3]
[6:23] <mosu001> pg 0.2 is stuck peering, last acting [9,3]
[6:23] <mosu001> pg 1.1 is stuck peering, last acting [9,3]
[6:23] <mosu001> pg 1.2ff is stuck peering, last acting [8,0]
[6:23] <mosu001> pg 2.2ff is stuck peering, last acting [9,2]
[6:23] <mosu001> pg 2.2fe is stuck peering, last acting [8,0]
[6:23] <mosu001> pg 0.2fa is stuck peering, last acting [8,0]
[6:23] <mosu001> pg 2.2f8 is stuck peering, last acting [8,0]
[6:23] <mosu001> pg 1.2f9 is stuck peering, last acting [8,0]
[6:23] <mosu001> pg 1.2ee is stuck peering, last acting [3,8]
[6:23] <mosu001> pg 0.2ef is stuck peering, last acting [3,8]
[6:23] <mosu001> ceph health detail | grep osd gives nothing, wouldn't that indicate a down osd?
[6:24] <mosu001> When I check for the osd I get
[6:24] <mosu001> osdmap e37 pg 1.c (1.c) -> up [0,8] acting [0,8]
[6:24] <mosu001> So primary is osd.0 and secondary is osd.8?
[6:25] <The_Bishop> seems like this
[6:25] <mosu001> I see osd.8 and osd.9 having the slow requests issues in ceph -w
[6:26] <The_Bishop> after re-formatting an osd, there may be heavy data copy action in the background
[6:26] <sage> are osd 0 and 8 up?
[6:27] <The_Bishop> are osd.8 and osd.9 the devices you did reformat?
[6:27] <sage> if pg query hangs, they are not responding for one reason or another
[6:27] <mosu001> I've got nmon looking at disk io on the OSD nodes and there is nothing happening... but I can leave it overnight?
[6:27] <mosu001> Actually I had to reformat all the disks... they all had superblock errors
[6:27] <mosu001> I did
[6:28] <The_Bishop> *ouch*
[6:28] <mosu001> mkfs.btrfs /dev/<disk>
[6:28] <mosu001> mount /dev/<disk> /data/<osd>
[6:29] <mosu001> ceph-osd -c /etc/ceph/ceph/conf -i <ods id> --mkfs <mon map options>
[6:30] <mosu001> Then stop ceph and start it again and all the superblock errors went away
[6:30] <The_Bishop> and you reformatted osd.0 up to osd.99?
[6:34] <mosu001> osd.0 to osd.11
[6:34] <mosu001> osd.0-5 on one server and osd.6-11 on another with replication to each server (i.e., basically mirrored across servers)
[6:35] <mosu001> This happened afer I tried to iozone to separate files in the root ceph directory from clients on multiple machines...
[6:36] <mosu001> The system seemed to hang so I stopped the test and upgraded to openSUSE 12.1 and ceph v0.47
[6:36] <The_Bishop> hmm, in this case you erased everything, and MON or MDS has old information because they ran when you reformatted the osds
[6:36] <mosu001> Somewhere in the process my OSDs didn't like it...
[6:36] <mosu001> So they're trying to rebuild info that isn't actually there?
[6:36] <The_Bishop> dunno
[6:37] <The_Bishop> i always shut down everything before i do ceph surgery
[6:37] <mosu001> On a smaller system with 3 OSDs (2 on one machine and 1 on another - spread equally among the 3) I had to do the same thing to the OSDs on the first machine, so I expcetd to lose data, but I don't seem to have lost any (as far as I can tell so far)
[6:38] <sage> sjust: there?
[6:38] <mosu001> Yeah... the data is not critical and I have a copy anyway so maybe I need to "clobber" the data on the OSDs...?
[6:39] <mosu001> I wanted to test side by side r/w to a ceph FS using iozone, but I've been told even though the files are different the directory inode may have been locking causing the "hang"
[6:41] <mosu001> I'm happy to "zap" the data just to get the system up again
[6:41] <mosu001> So I could take ceph down and erase the partitions to be raw again and then let ceph rebuild the btrfs from scratch?
[6:43] <mosu001> Soory guys, gotta do a child-care pick up. Will check back in a few hours so if you've got any suggestions, I'd love to hear them
[6:43] <The_Bishop> well, it is good to have any ceph processes stopped while doing wetwork
[6:44] <mosu001> By the way, I've been looking at the failures page and it does have good advice, but nothing that has helped here
[6:44] <The_Bishop> it's experimental (tm) *smile*
[6:45] <mosu001> Coincidentally, much of my ceph "bending" is happening while benchmarking and also comparing to OpenStack's swift
[6:46] <mosu001> Yep, I usually stop all the ceph processes, but had it up while restoring the OSDs to get the monmap... Will try the partition idea later on if no better suggestions arise
[6:46] <sage> mosu001: sorry, not fully paying attention, but: the change from before is that ceph-osd --mkfs doesn't wipe the data directory for you; you need to do that yourself.
[6:46] <sage> you probably have the old cluster's osd data there, and it probably refuses to talk to the clsuter because the cluster fsid is different
[6:46] <sage> wipe out hose directories and re-run mkcephfs and you should be set
[6:46] <mosu001> Oh ok
[6:46] <The_Bishop> that seems probable
[6:47] <mosu001> I was only trying to fix the superblock problem, does that mean data must go too
[6:48] <mosu001> Also, by those directories you mean the osd directories right? In my case /data/<osd>
[6:48] <The_Bishop> umount /ceph/osd.$i ; mkfs.btrfs -m single -d single /dev/mapper/osd.$i ; mount /dev/mapper/osd.$i /ceph/osd.$i ; ceph-osd --mkfs -i $i
[6:49] <mosu001> Thanks, can I try just for the hanging osd first
[6:49] <mosu001> And then all osds if problems persist?
[6:50] <The_Bishop> yes, replication will kick in when the failing osd is "replaced"
[6:50] <mosu001> Awseom, I'll give it a shot tonight
[6:50] <mosu001> Thanks sage, thanks The_Bishop
[6:50] * The_Bishop bows to mosu001
[6:51] * renzhi (~renzhi@ Quit (Read error: Operation timed out)
[7:13] * chutzpah (~chutz@ Quit (Quit: Leaving)
[7:25] * renzhi (~renzhi@ has joined #ceph
[7:43] * The_Bishop (~bishop@cable-86-56-102-91.cust.telecolumbus.net) Quit (Remote host closed the connection)
[7:55] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) has joined #ceph
[8:18] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) Quit (Quit: Leaving.)
[8:32] * The_Bishop (~bishop@e177089186.adsl.alicedsl.de) has joined #ceph
[8:56] * renzhi (~renzhi@ Quit (Ping timeout: 480 seconds)
[9:04] * BManojlovic (~steki@ has joined #ceph
[9:06] * renzhi (~renzhi@ has joined #ceph
[9:08] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:09] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Remote host closed the connection)
[9:12] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:16] * loicd (~loic@ has joined #ceph
[9:40] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[11:15] * s[X]__ (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[11:15] * s[X]_ (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Read error: Connection reset by peer)
[11:42] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[12:25] * renzhi (~renzhi@ Quit (Quit: Leaving)
[12:53] * Ryan_Lane (~Adium@p5DDC6B18.dip.t-dialin.net) has joined #ceph
[12:59] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[13:04] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[13:46] * The_Bishop_ (~bishop@e179010000.adsl.alicedsl.de) has joined #ceph
[13:53] * The_Bishop (~bishop@e177089186.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[14:18] * thafreak (~thafreak@ Quit (Remote host closed the connection)
[14:21] * s[X]__ (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Ping timeout: 480 seconds)
[14:21] * thafreak (~thafreak@ has joined #ceph
[14:21] * s[X] (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[15:16] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[15:52] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[16:02] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[16:21] * The_Bishop_ (~bishop@e179010000.adsl.alicedsl.de) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[16:29] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[16:41] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[17:01] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:05] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:28] * goddamnSteveJobs (~goddamnSt@09GAAGFWQ.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:46] * aliguori (~anthony@cpe-70-123-145-39.austin.res.rr.com) Quit (Quit: Ex-Chat)
[17:59] * BManojlovic (~steki@ has joined #ceph
[18:02] * yehudasa_ (~yehudasa@aon.hq.newdream.net) has joined #ceph
[18:05] * OutBackDingo (~quassel@rrcs-71-43-84-222.se.biz.rr.com) Quit (Remote host closed the connection)
[18:07] * Tv_ (~tv@aon.hq.newdream.net) has joined #ceph
[18:07] * Tv_ is now known as tv_
[18:07] <sage> gregaf: when you get in, can you look at the last few patches of wip_rolling_upgrade (dealing with monitor compatibility)?
[18:07] <goddamnSteveJobs> gregaf: when you get in, can you look at the last few patches of wip_rolling_upgrade (dealing with monitor compatibility)?
[18:08] * OutBackDingo (~quassel@rrcs-71-43-84-222.se.biz.rr.com) has joined #ceph
[18:08] <sage> we neglected to deal with this when we did the encoding changes
[18:08] <goddamnSteveJobs> we neglected to deal with this when we did the encoding changes
[18:09] <elder> Well that's interesting.
[18:09] <goddamnSteveJobs> Well that's interesting.
[18:10] <elder> Can we get rid of goddamnSteveJobs somehow?
[18:10] <goddamnSteveJobs> Can we get rid of goddamnSteveJobs somehow?
[18:10] * tjfontaine (tjfontaine@tjfontaine.ombudsman.oftc.net) has joined #ceph
[18:10] * tjfontaine sets mode +o tjfontaine
[18:11] <sage> go away, steve jobs
[18:11] <goddamnSteveJobs> go away, steve jobs
[18:11] * yehudasa /test
[18:11] <elder> Maybe tjfontaine can help.
[18:11] <goddamnSteveJobs> Maybe tjfontaine can help.
[18:11] <tjfontaine> oh an echo bot, of course
[18:11] <goddamnSteveJobs> oh an echo bot, of course
[18:11] * goddamnSteveJobs (~goddamnSt@09GAAGFWQ.tor-irc.dnsbl.oftc.net) Quit (Killed (tjfontaine (No reason)))
[18:11] <elder> Yay!
[18:11] * zobel (zobel@zobel.noc.oftc.net) has joined #ceph
[18:11] <zobel> mh?
[18:11] <sage> tjfontaine: thanks!
[18:11] <tjfontaine> zobel: another echo bot
[18:12] <zobel> ah.
[18:12] <nhm> tjfontaine: thanks!
[18:12] * zobel goes back to sleep.
[18:12] <tjfontaine> sage: so to take care of chanops we need a way to determine who should get the initial master
[18:12] * zobel suggests: tj
[18:12] * elder changes NI to goddamSteveJobs :)
[18:13] <sage> tjfontaine: we've been using this channel for several years now.. see http://ceph.com/resources/mailing-list-irc/
[18:13] <elder> sage should get the initial master, tjfontaine
[18:13] <tjfontaine> sage: I'm aware, and I dropped in a while ago and mentioned we should fix the fact that its not registered :)
[18:13] <tjfontaine> ok
[18:13] <sage> tjfontaine: ah :) sorry, i missed it apparently :)
[18:13] <nhm> tjfontaine: Yeah, give it to Sage. :)
[18:13] * tjfontaine sets mode +o sage
[18:14] <elder> All hail!
[18:14] <yehudasa> whoa!
[18:14] <elder> Now you can set the channel topic
[18:14] <nhm> By the power of grayskull!
[18:14] <sage> tjfontaine: thanks!
[18:14] <tjfontaine> might I also suggest http://oftc.net/oftc/NickServ/CertFP for those users who are soon to be registering
[18:15] <sage> i'll take a look
[18:16] * tjfontaine sets mode -o tjfontaine
[18:16] <sage> it may be time for me to abandon pidgin for irc
[18:16] <tjfontaine> oh god, PLEASE
[18:16] * sage was expecting that
[18:16] <tjfontaine> libpurple has the worst client library
[18:17] * zobel (zobel@zobel.noc.oftc.net) has left #ceph
[18:17] <nhm> sage: I tried that for a couple of days, but ended up with good old irssi again.
[18:17] <tjfontaine> anyway, I'll lurk for a little while and make sure things are ok
[18:18] <jmlowe> a irc client must flap at me indicating new activity before I will consider it
[18:18] * cattelan_away_away_away_away is now known as cattelan_away_away_away
[18:18] * aliguori (~anthony@ has joined #ceph
[18:19] * ChanServ sets mode +o sagewk
[18:20] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[18:23] * mtk00 (eTK4IETzge@panix2.panix.com) has joined #ceph
[18:23] <sage> aha: root@plana50:~# apt-get -y install mencoder=2:1.0~rc4.dfsg1+svn34540-1
[18:23] <sage> E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
[18:23] <sage> dmick: let's add 'dpkg --configure -a' to the chef sequence?
[18:24] <nhm> sage: strange that it was interrupted in the first place though.
[18:24] <sage> or nhm, whoever is around?
[18:24] <sage> yeah... but we should tolerate it either way
[18:26] <nhm> would that just be an execute statement at the beginning of the default recipe? I confess I don't know chef that well.
[18:27] <sage> beats me :)
[18:28] * mtk0 (~mtk@ool-44c35bb4.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[18:28] <sage> can wait for dmick i guess. the one that screwed the qa runs last night is fixed at least
[18:43] * mtk00 (eTK4IETzge@panix2.panix.com) Quit (Remote host closed the connection)
[18:44] * mtk00 (~mtk@ool-44c35bb4.dyn.optonline.net) has joined #ceph
[18:49] <joao> I should create an utility company to provide coffee on the tap of every home
[18:50] <joao> hate when I run out of coffee
[18:50] <gregaf> that sounds disgusting
[18:50] <joao> I don't care
[18:50] <joao> as long as it contains caffeine and has a similar taste to coffee, I'd buy it
[18:51] <gregaf> I feel like a coffee delivery service would work better
[18:51] <nhm> gregaf: Depends how it's done I think.
[18:51] <joao> gregaf, that would take way too long
[18:51] <gregaf> not if they brought you a (or many) thermos every morning
[18:52] <joao> as long as the coffee was kept hot all day long, that could work
[18:52] <joao> but I like the sound of having a tap with hot coffee all day long
[18:52] <elder> I think Joao you didn't spend enough time in LA to appreciate the important difference between good coffee and "similar taste to coffee."
[18:53] <nhm> Is LA known for good coffee?
[18:53] <joao> elder, dreamhost and their espresso machine are to blame on that subject :p
[18:54] <elder> Apparently it's known for coffee better than what's found in Portugal.
[18:54] <nhm> We usually buy at costco, but every once and a while order direct from Costa Rica as a treat.
[18:54] <tv_> nhm: dunno about coffee but LA is pretty.. food-snobby
[18:54] <elder> joao, do you think your tap water is fine as long as it has a "similar taste to water?"
[18:54] <elder> Maybe your tap water tastes like coffee.
[18:55] <joao> elder, of course not
[18:55] <nhm> Minneapolis tap water is actually pretty good because they have to filter it.
[18:55] <joao> the point being, I ran out of coffee, and anything with a similar taste would work right now
[18:55] <joao> I do enjoy good coffee though
[18:56] <joao> I'm a bit of a snob on that regard; aside from when I run out of it
[18:56] <joao> besides, I could provide a premium service, with coffee with coffee taste, and a regular service, with coffee with similar taste :p
[18:57] <elder> And maybe "coffee" for which the taste doesn't matter at all.
[18:58] <gregaf> I'm just saying, coffee isn't like water ??? it has stuff in it
[18:58] <gregaf> sending it through pipes? eww
[18:59] <joao> gregaf, depends on the distance and how well maintained those pipes are, I suppose
[18:59] <joao> when I was a kid, downtown Lisbon had a system of beer pipes for the big restaurants
[19:01] <joao> I have no idea how that used to work, all I know if from what my parents told me
[19:01] <sjust> seems like you could just add caffeine to the water
[19:02] <elder> Or "brown"
[19:04] * chutzpah (~chutz@ has joined #ceph
[19:06] <gregaf> sagewk: after waffling around it for a bit, that monitor stuff looks good
[19:06] <gregaf> assuming you correctly copied the encoding from the previous versions; I didn't check that
[19:12] <sagewk> i cut and pasted, and tested with vstart, and all seemed well.
[19:13] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[19:13] <sagewk> gregaf: the other annoying transition with this version is ceph -s/-w.. ceph and ceph-mon need to match for it to work. hopefully nobody is scripting against those
[19:14] <gregaf> hrm, yeah
[19:35] <dmick_away> sage, nhm: about dpkg --configure: jeez, how many other ways do we have to wipe apt's butt for it
[19:35] <dmick_away> what's the point of saying "oo things are bad, you should do this specific cleanup" and then *NOT DOING THE CLEANUP*. Grumble.
[19:36] * dmick_away is now known as dmick
[19:44] <tv_> "dpkg --configure --pending" is the usual thing
[19:45] <tv_> -a is almost the same
[19:45] <tv_> but at that point, it's like.. you're already fucked, i don't trust automatic recovery much
[19:45] <dmick> presumably this repairs interrupted package install operations, and is otherwise safe-ish?
[19:45] <dmick> other than the "we are now entering WTF land" part
[19:45] <tv_> well, it'll retry
[19:45] <tv_> if something is really broken, retry isn't enough
[19:46] <dmick> true. but it'll mop up some simple interruption half-state things. eh.
[19:46] <dmick> one wonders why apt doesn't just do this
[19:46] <tv_> because it's a bug in the first place
[19:46] <tv_> hiding bugs is baaad
[19:47] <dmick> well, so, if apt-get install is interrupted by something it can't catch, is that really a bug?...
[19:47] <tv_> dpkg is not transaction safe
[19:47] <dmick> right
[19:47] <tv_> you just went off to unknown-land
[19:47] <tv_> do not interrupt dpkg lightly
[19:47] <sagewk> gregaf: before you shift into chef mode, can you review wip-2022 (just the last patch)?
[19:47] <dmick> do not taunt happy fun ball
[19:48] <nhm> I suppose one question is whether or not running dpkg/apt-get from a teuthology task is a good idea in the first place.
[19:48] <dmick> ok. this is leading me toward "add the damn cleanup command"
[19:48] <nhm> apt/dpkg do *not* like being iterrupted.
[19:48] <nhm> tv_: sorry, you just said the same thign.
[19:48] <tv_> dmick: i will argue you're just asking to be hurt more
[19:49] <tv_> dmick: why did it fail
[19:49] <dmick> lost in the mists of time
[19:49] <tv_> that's not a very good answer :-/
[19:49] <dmick> "I'm the cleanup crew for a party I was too young to attend"
[19:49] <morpheus> sagewk: talking about http://tracker.newdream.net/issues/2602 - is their any way to see if the metadata got corrupted? cluster is running fine without problems since my last update try
[19:49] <tv_> dmick: this is why i'm advocating frequent reinstalls
[19:49] <dmick> I hear you
[19:49] <dmick> and I'm reticent to add the cleanup for all the same reasons
[19:50] <dmick> but I'm also a pragmatist :
[19:50] <dmick> )
[19:50] <sagewk> morpheus: oh, that's good. the daily scrub should catch anything
[19:51] <tv_> dmick: well it's like.. there's this story on the web somewhere of a guy setting up a roomba to clean his apt as he leaves for the day
[19:51] <tv_> dmick: in the meanwhile, his dog poops on the floor
[19:51] <tv_> dmick: naturally, roomba proceeds to smear the poop all over the apt
[19:51] <morpheus> no errors so far, i'll try the wip_rolling_upgrade soon
[19:51] <tv_> dmick: i do not believe in automated cleanup of exceptional situations, when it's not in the original design
[19:51] <sagewk> morpheus: if you've already upgraded, the rolling upgrade branch won't do anything new
[19:52] <tv_> dmick: sure, your roomba may pick up some of the popcorn spilled at the party last night, but one day you'll come home to see a bigger mess
[19:52] <nhm> tv_: dmick: user-driven installs would be nice. IE you lock nodes, and then it's up to you to put an image on them.
[19:52] <dmick> tv_: again, philosophically I'm with you
[19:52] <sagewk> tv_,dmick: works for me. let's just note this is one reason for the bazillion chef failures in the future.
[19:52] <morpheus> sagewk: still on 0.47-2, i took the broken next branch osd out of the cluster
[19:52] <tv_> nhm: way ahead of you -- you'll request nodes with a certain install on them
[19:52] <nhm> tv_: sounds peachy
[19:52] <dmick> and am happy to find dpkg.log to perhaps part some of those time mists; lookingg
[19:52] <sagewk> morpheus: oh, cool. in that case, testing wip_rolling_upgrade would be great :)
[19:53] <sjust> morpheus: any errors would be restricted to the upgraded osd
[20:01] * Ryan_Lane1 (~Adium@ has joined #ceph
[20:07] * Ryan_Lane (~Adium@p5DDC6B18.dip.t-dialin.net) Quit (Ping timeout: 480 seconds)
[20:14] * loicd (~loic@ Quit (Ping timeout: 480 seconds)
[20:16] <gregaf> sagewk: sorry for the delay ?????looks fine to me, but I think I tried something similar to this and had either joshd or sjust nix it, let me check with them
[20:17] <sjust> gregaf: what?
[20:17] <gregaf> wip-2022
[20:17] <gregaf> no longer let send_linger() independently recalc_op_target(), but just do it in the op_submit() function
[20:20] <dmick> I don't see anything smoking in dpkg.log. ?
[20:20] <dmick> let's see how often we get screwed here.
[20:21] <sjust> wip-2022 looks good to me if it's been tested
[20:22] * The_Bishop (~bishop@2a01:198:2ee:0:1442:c2de:444c:a36a) has joined #ceph
[20:23] <gregaf> okay, apparently I made it up, so wip-2022 looks good sagewk!
[20:23] <sagewk> thanks
[20:26] * Ryan_Lane1 (~Adium@ Quit (Quit: Leaving.)
[20:31] <sagewk> tv_: latest upstart-vs-logrotate?
[20:32] <tv_> ahahah
[20:33] <tv_> still trying to parse that mawk
[20:35] <tv_> i don't like the '$type' thing very much, and the gsubs seem unnecessary, but hold on..
[20:40] <tv_> dammit mawk has no capturing of regex groups
[20:42] <dmick> just the one, &, right?
[20:43] <tv_> & is all of the match, at least in sed
[20:43] <tv_> i want to extract subparts
[20:43] <tv_> writing perl ')
[20:43] <tv_> ;)
[20:44] <dmick> all of the match of the group. I guess I don't know for sure what happens if you have multiple groups
[20:45] <tv_> echo 'ceph-mon (ceph/foo) start/running'|perl -ne 'print "$+{service} cluster=$+{cluster} id=$+{id}\n" if m{^(?<service>ceph-\S+)\s+\((?<cluster>[^/)]+)/(?<id>[^)]+)\)}'
[20:45] <tv_> okay tighten up a little and then it's good
[20:47] <tv_> sagewk: http://pastebin.com/mnP4ng7m
[20:47] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:47] <tv_> ohh make that initctl reload -- $l 2>/dev/null || :
[20:47] <tv_> just because i'm paranoid ;)
[20:55] * m5jasonb (~jason@lists.command-line.info) has joined #ceph
[20:57] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:57] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:00] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[21:06] * aliguori (~anthony@ Quit (Ping timeout: 480 seconds)
[21:12] * Ryan_Lane (~Adium@dslb-094-223-088-003.pools.arcor-ip.net) has joined #ceph
[21:20] * mtk00 (~mtk@ool-44c35bb4.dyn.optonline.net) Quit (Quit: Leaving)
[21:21] * mtk (~mtk@ool-44c35bb4.dyn.optonline.net) has joined #ceph
[21:38] <sagewk> tv_: thanks
[21:51] * aliguori (~anthony@ has joined #ceph
[22:12] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[22:30] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:34] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[22:34] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:41] <joao> \o/
[22:43] <dmick> goal?
[22:45] <elder> No, Goooooooooooooooooooooooooooooooooooooooooooooooooooaaaaal!!!!!
[22:51] <mosu001> ceph health
[22:51] <darkfader> ceph is very healthhy
[22:51] <mosu001> Hi again everyone, I am still having aproblem with ceph hanging due to pgs peering
[22:52] <mosu001> (Sorry, had wrong window selected)
[22:52] <mosu001> The_Bishop and sage helped last night, but still no luck
[22:52] <jmlowe> anybody care to elaborate on the fiemap ioctl problem?
[22:53] <mosu001> I have deleted the directories of my OSDs, used mkfs.btrfs, mkcephsfs and still the system hangs with pgs pering even though there is no disk io
[22:54] <mosu001> I see "slow request" for osd_op a lot, but if I re-start ceph then this slow request "moves" to a new OSD...
[22:54] <mosu001> At this stage I'm wondering if I should try an older version of ceph to see if that helps...
[22:54] <joshd> jmlowe: fiemap returned different results on the same fd with no writes in between, even though the fd was already fsync'd before calling fiemap
[22:54] <mosu001> Anyone with any ideas?
[22:54] * tjfontaine (tjfontaine@tjfontaine.ombudsman.oftc.net) has left #ceph
[22:55] <joshd> jmlowe: I haven't found a kernel that doesn't have this problem yet for xfs and btrfs
[22:55] <jmlowe> joshd: specific to one filesystem type or broken across the board?
[22:55] <jmlowe> joshd: read my mind
[22:56] <jmlowe> joshd: ext4 ok or untested?
[22:56] <joao> dmick, we won :D
[22:56] <joshd> jmlowe: ext4 seems to work, but I'm not sure if it's just harder to hit the race there on our test hardware
[22:56] <joao> going to the semi-finals, it seems
[22:57] <jmlowe> joshd: is it a race condition, only happens under load?
[22:57] <dmick> joao: are you on a team?
[22:58] <joao> I'm on team Portugal :p
[22:58] <joshd> jmlowe: it seems to be - if you do the fiemap later you get the correct result
[22:59] <joshd> jmlowe: I haven't tested without load
[22:59] <jmlowe> joshd: wonder if that hasn't been causing me problems, had several rbd's go bad while some others seem ok, seemed to be ones with heavier writes
[23:00] <joshd> jmlowe: it's certainly possible. can't really diagnose after the fact though
[23:03] <joshd> it would be load on the osd, not the guest, if it were an issue
[23:03] <jmlowe> osd's are always loaded
[23:04] <jmlowe> was also hit by a leveldb bug and some watchdog rebooting, really kind of a perfect storm
[23:05] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[23:11] * aliguori (~anthony@ Quit (Remote host closed the connection)
[23:39] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:39] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:42] * s[X] (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Remote host closed the connection)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.