#ceph IRC Log

Index

IRC Log for 2012-11-13

Timestamps are in GMT/BST.

[0:02] <slang> sagewk: just pushed wip-3431
[0:03] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[0:03] <slang> sagewk: not sure if that's the right fix, but it does fix the assert failure
[0:06] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) Quit (Quit: Leaving.)
[0:12] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[0:13] * drokita (~drokita@199.255.228.10) Quit (Ping timeout: 480 seconds)
[0:13] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit ()
[0:16] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) has joined #ceph
[0:16] <jefferai> joshd1: have you seen a "PermissionsError: error unprotecting snapshot <uuid>@snap" when doing a glance image-delete?
[0:17] <jefferai> at first I thought I had this:
[0:17] <jefferai> https://bugs.launchpad.net/glance/+bug/1075580
[0:17] <jefferai> but I applied the patch, and then realized that the error wasn't a typeerror, it was a permission error
[0:21] <joshd1> jefferai: what osd caps does the glance user have?
[0:22] <jefferai> [mon] allow r [osd] allow rwx pool ssds
[0:23] <jefferai> when I run the command, I see on osd.5.log "CephxAuthorizeHandler::verify_authorizer isvalid=1"
[0:23] <jefferai> not sure if tha means it's successful or not, although it would seem the former
[0:24] <jefferai> not sure if this matters, but it seems to only happen with a bare container
[0:24] <joshd1> container type shouldn't matter
[0:24] <jefferai> actually, I guess the difference is that with the container not set it's in "queued" status
[0:24] <jefferai> not active
[0:24] <jefferai> agreed, but if you don't set it you get a 400 error and the status is queued...
[0:25] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[0:26] <joshd1> does the image and snapshot show up in rbd ls -p ssds -l?
[0:27] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[0:27] <jefferai> I see entries with uuids matching what's in glance-list
[0:27] <jefferai> I don't see any snapshots
[0:27] <jefferai> (I also haven't, at least, manually specified any snapshots to be created)
[0:28] <joshd1> were these created with an earlier version of librbd?
[0:28] <jefferai> nope
[0:28] <jefferai> they were added a few commands earlier when following http://docs.openstack.org/trunk/openstack-compute/install/apt/content/uploading-to-glance.html
[0:28] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[0:28] <joshd1> the rbd store in glance will create and protect a snapshot after uploading the data
[0:29] <joshd1> if there are no snapshots left, they must have been unprotected and removed, or never existed in the first place
[0:29] <jefferai> I don't think they ever existed
[0:29] <jefferai> although
[0:29] <jefferai> let me try adding again
[0:29] <jefferai> and see if they exist before I run image-delete the first time
[0:30] <jefferai> I should see the snapshots with rbd ls -ps ssds -l?
[0:30] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) Quit (Quit: Leaving.)
[0:30] <joshd1> yeah
[0:30] <joshd1> unless that wasn't in 0.53...
[0:31] <jefferai> well
[0:31] <jefferai> if I run rbd -p ssds snap ls <uuid> then I see one thing back
[0:31] <jefferai> with an id, a name of "snap"
[0:31] <lurbs> -l isn't in the 0.53 in the debian-testing repository.
[0:31] <jefferai> ah, so maybe it's being ignored
[0:31] <jefferai> and just doing the ls
[0:32] <jefferai> but doing the "snap ls" seems to indicate that there is indeed a snapshot
[0:33] <joshd1> and does 'rbd info <uuid>@snap' show that it's protected?
[0:34] * noob2 (47f46f24@ircip3.mibbit.com) has joined #ceph
[0:34] <jefferai> give me an error
[0:34] * BManojlovic (~steki@85.222.180.134) Quit (Remote host closed the connection)
[0:34] <jefferai> "error opening image <uid>: (2) No such file or directory"
[0:34] <noob2> can ceph handle having rbd's mounted back on itself?
[0:34] <jefferai> but it does show up in rbd ls
[0:35] <jefferai> that uuid I mean
[0:35] <jefferai> and it is valid in "snap ls"
[0:35] <jefferai> ah wait
[0:35] <jefferai> sorry
[0:35] <jefferai> I bet I need to use -p ssds
[0:35] <jefferai> sec
[0:35] <jefferai> nope
[0:35] <jefferai> didn't change anything
[0:35] <jefferai> wait
[0:35] <jefferai> ok
[0:36] <jefferai> joshd1: so, running "rbd -p ssds info <uuid>" works
[0:36] <jefferai> but running "rbd -p ssds info <uuid>@snap" does not, gives an error
[0:36] <joshd1> but "rbd snap ls -p ssds uuid" shows a single snapshot named 'snap'?
[0:37] <joshd1> that would be very strange
[0:37] <jefferai> here's the output I get:
[0:37] <jefferai> ID NAME SIZE
[0:37] <jefferai> 5 snap 9761280
[0:38] <sagewk> joshd1: can you peek at wip-oc-enoent?
[0:39] <jefferai> joshd1: http://paste.kde.org/605180/raw/
[0:41] <joshd1> sagewk: looks good
[0:41] <sagewk> k thanks
[0:42] <sagewk> slang: the first patch looks good, altho you can do set_item.remove_myself() instead of working via the containing list
[0:43] * loicd (~loic@93.158.30.16) Quit (Ping timeout: 480 seconds)
[0:43] <sagewk> for the second patch: basically all the other stuff in can_close() does not imply lru_expireable, but the reverse is true?
[0:43] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[0:43] <sagewk> i think a better fix is to kill can_close() entirely (it's a holdover from the old regime)
[0:44] <sagewk> is this a reproducible bug?
[0:44] <slang> sagewk: yeah
[0:45] <slang> sagewk: emailed you the test I was using to reproduce, with the delay stuff enabled
[0:46] <slang> sagewk: I'm off to pet therapy - will be back online later
[0:47] <joshd1> jefferai: could you pastebin the output of 'rbd info -p <uuid>@snap --debug-ms 1 --debug-rbd 20'?
[0:47] <jefferai> yeah
[0:48] <sagewk> slang: k thanks
[0:48] <jefferai> joshd1: http://paste.kde.org/605192/raw/
[0:48] * yanzheng (~zhyan@101.83.212.151) has joined #ceph
[0:50] <ghbizness> looks like now that we modified the crush table and it looks good,,, we now have root@csn2:~# ceph health
[0:50] <ghbizness> HEALTH_WARN 570 pgs stuck unclean
[0:50] <ghbizness> root@csn2:~#
[0:50] <jefferai> joshd1: interesting -- it's considering the whole string part of the name, not 'snap' as the snap name
[0:50] <jefferai> line 19
[0:51] <ghbizness> what would be the best way to get the stuck blocks unstuck
[0:51] <jefferai> ghbizness: WD-40
[0:51] <joshd1> jefferai: yeah, that's odd... it's working in master, and I don't remember any bugs like that fixed
[0:51] <ghbizness> lol, i prefer silicone grease
[0:51] <ghbizness> cleaner
[0:52] <ghbizness> lasts longer
[0:52] <joshd1> jefferai: in any case, try 'rbd info -p ssds <uuid> --snap snap'
[0:53] <joshd1> ghbizness: what's osd tree and ceph -s now?
[0:53] <jefferai> joshd1: that works fine
[0:53] <jefferai> and does show it's protected
[0:54] <ghbizness> http://pastebin.com/4UCVjd1T
[0:54] <joshd1> ok, strange that you found a previously unknown (and fixed?) parsing bug
[0:54] <ghbizness> overall, the OSD tree looks good
[0:54] <noob2> in the future will ceph packages be coming out for rhel5 or is that a forgotten relic now :D
[0:55] <jefferai> joshd1: hah, happy to help
[0:55] <jefferai> do you think that openstack is using the invalid syntax, then?
[0:55] <joshd1> no, glance is using librbd directly
[0:56] <joshd1> and a PermissionsError is totally different as well
[0:56] <ghbizness> please note that we added racks abd moved hosts from one rack to another
[0:56] <jefferai> sure
[0:56] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) Quit (Quit: ...)
[0:56] <ghbizness> abd = and
[0:58] * yanzheng (~zhyan@101.83.212.151) Quit (Ping timeout: 480 seconds)
[0:59] <joshd1> jefferai: well, the only places EPERM can come from are the osd and monitor
[0:59] <jefferai> ok, I'll watch logs
[0:59] <joshd1> jefferai: try changing the osd caps for the glance client to just rwx on anything
[0:59] <jefferai> so I need to watch the logs on each individual box?
[0:59] <jefferai> or is there some wya to know which boxes to watch?
[1:00] <joshd1> no, it wouldn't show up in the logs unless you had lots of debugging on anyway
[1:00] <jefferai> ah
[1:00] <joshd1> you could enable client side logging for glance (debug ms = 1, debug auth = 20, debug rbd = 20, log file = /path/writable/by/glance)
[1:01] <joshd1> ghbizness: did you modiy the crush rules at all?
[1:02] <ghbizness> not the rules
[1:03] <ghbizness> 1sec, making a diff of old and new
[1:03] <joshd1> ghbizness: could you pastebin an osd dump and a 'ceph pg dump_stuck unclean'?
[1:03] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[1:03] <jefferai> joshd1: I now have mon allow rwx and osd allow rwx pool ssds
[1:03] <jefferai> and still fails
[1:03] <joshd1> jefferai: try without the 'pool ssds' part
[1:04] <jefferai> joshd1: hurrah
[1:05] <joshd1> argh, that means the osd cap parsing broke even worse in 0.53
[1:05] <jefferai> doesn't work with pool=ssds either
[1:05] <jefferai> but that's known
[1:05] <joshd1> yeah
[1:05] <joshd1> all of those syntaxes work with the next branch
[1:06] <jefferai> when does that get into debian-\testing?
[1:06] <joshd1> I think the end of the week
[1:06] <jefferai> okay
[1:07] <jefferai> or are nightlies in a really good state and I should move to them?
[1:07] <jefferai> scares me a bit to do that, as this is for production
[1:07] <ghbizness> joshd1, http://pastebin.com/GyG7fjR8
[1:08] <joshd1> the next branch doesn't change that much, it's been mostly frozen for a few weeks now
[1:08] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit (Quit: Leaving.)
[1:08] <joshd1> it's the master branch that'll break out from under you
[1:09] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[1:09] <jefferai> ah, I thought the master branch was pretty solid and was the basis for debian-testing
[1:09] <jefferai> and only got merged into after testing
[1:09] <jefferai> guess I had it backwards
[1:10] <joshd1> yeah, the testing branch is the last development release (i.e. debian-testing)
[1:10] <jefferai> ah, k
[1:11] <jefferai> for now I'm fine keeping the permissions open
[1:11] <jefferai> on the next update I'll try again
[1:12] <iggy> shame ceph got booted from debian's next release, but i guess ubuntu is probably a better fit for ceph anyways
[1:14] <joshd1> ghbizness: ok, so I'm guessing this is a bug - the pg temp overrides are supposed to get cleaned up once recovery is done
[1:14] * noob2 (47f46f24@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[1:14] <ghbizness> nice... 3rd bug today
[1:15] <joshd1> ghbizness: one thing to check though - are your monitor's times in sync?
[1:15] <ghbizness> btw... http://pastebin.com/PvknFZYq
[1:15] <ghbizness> yes, NTP is up on all nodes
[1:15] * noob2 (a5a00214@ircip4.mibbit.com) has joined #ceph
[1:15] <ghbizness> i can check the time again
[1:15] <noob2> do the ceph rhel6 rpm's not include the rbd module?
[1:15] <ghbizness> the above pastbin is a diff of the old and new crush map file
[1:16] <iggy> noob2: i'd imagine there was a rbd-kmod package or something
[1:16] <noob2> ok
[1:16] <noob2> checking around for it
[1:16] <noob2> ah cool. yeah these work right out of the box
[1:18] <joshd1> ghbizness: your rack weights are all 4.0, but each only has 1.0 from each of two osds?
[1:18] <noob2> iggy: i'll have to take a look later. i'm having trouble finding it :(
[1:18] <ghbizness> yes, im guessing that should be 2.0 ?
[1:19] <ghbizness> is there a set rule on how to set the weights for the racks ?
[1:19] <joshd1> noob2: I don't think there are packages for rhel6 for the kernel module
[1:20] <joshd1> ghbizness: the sum of the weights underneath them
[1:20] <joshd1> noob2: rhel6 still has a heavily patched 2.6.32 iirc, which is really old
[1:21] <iggy> mmm, yeah, there's that too
[1:25] <ghbizness> joshd1, that cleared up 4/5 of the stuck uncleans
[1:25] <ghbizness> health HEALTH_WARN 123 pgs stuck unclean
[1:25] <ghbizness> one thing to NOTE...
[1:25] <ghbizness> device0 = osd.0
[1:26] <ghbizness> both time that i imported the crush map, osd.0 showed as in and up but was not under any rack or host
[1:26] <ghbizness> or pool
[1:27] * jlogan1 (~Thunderbi@72.5.59.176) Quit (Read error: Connection reset by peer)
[1:27] <joshd1> ghbizness: you might also try setting the crush tunables http://ceph.com/docs/master/cluster-ops/crush-map/#tunables
[1:27] * jlogan1 (~Thunderbi@72.5.59.176) has joined #ceph
[1:27] <ghbizness> http://pastebin.com/f4qQPLTU
[1:27] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) has joined #ceph
[1:28] <sagewk> slang: take a look at the updated wip-3431
[1:31] <ghbizness> joshd1, you the man...
[1:31] <ghbizness> # ceph health
[1:31] <ghbizness> HEALTH_OK
[1:32] <joshd1> awesome
[1:32] <ghbizness> josh, u in newark ?
[1:33] <joshd1> no, that's just the random irc server I was routed through
[1:33] <joshd1> los angeles
[1:34] <ghbizness> got it
[1:34] <ghbizness> i saw that... and was too late.. already asked
[1:34] <ghbizness> South Florida here
[1:35] <noob2> joshd1: i was afraid of that haha. so is the fuse client the way to go then with rhel6?
[1:35] * maxiz (~pfliu@111.192.251.7) Quit (Ping timeout: 480 seconds)
[1:35] <ghbizness> anyway, i am out for now.. leaving office. josh and all else, thank you
[1:36] <joshd1> noob2: for cephfs, yeah, unless you can upgrade your kernel
[1:36] <noob2> yeah prob can't upgrade the kernel. i'm looking to just use the rbd at this point
[1:37] <joshd1> ghbizness: see you, glad everything worked out eventually
[1:38] * maxiz (~pfliu@111.192.251.7) has joined #ceph
[1:38] <joshd1> noob2: for virtualization with qemu/kvm, or something else?
[1:38] <noob2> for mounting rbd's as block storage on physical boxes
[1:38] * bchrisman (~Adium@108.60.121.114) Quit (Quit: Leaving.)
[1:39] <noob2> if that won't work i was thinking of installing a fibre card on each osd server and mounting the rbd's locally and exporting over LIO
[1:39] <joshd1> generally kernel clients and osds on the same node aren't recommended
[1:39] <noob2> darn
[1:39] <noob2> so i'd need a proxy machine
[1:39] <joshd1> they can have a deadlock under memory pressure
[1:39] <joshd1> yeah
[1:40] <noob2> ok
[1:40] <noob2> that'll drive up the cost a little
[1:40] <noob2> i wonder if i could hijack a blade in the datacenter that is already fibre attached
[1:40] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) Quit (Quit: Leaving.)
[1:41] <joshd1> for any kernel client, you may need to upgrade the kernel
[1:41] <noob2> yeah
[1:42] <joshd1> there were a bunch of important networking fixes in 3.6
[1:42] <noob2> silly redhat and their ancient kernels
[1:42] <joshd1> there's a stable branch based on 3.5 too, but either way much newer than most distros
[1:42] <noob2> right
[1:43] <noob2> alright i'll have to price a proxy into the mix. we're all redhat 5/6 at my datacenter
[1:44] <noob2> they're also talking about using ceph rbd's for vmware which would require LIO anyways
[1:47] <iggy> interesting project... rbd for vmware
[1:47] <noob2> yeah the wheels are turning in managers heads
[1:47] <noob2> we pay insane prices for block storage to back vmware
[1:47] <noob2> over 10K a TB
[1:48] <noob2> i pitched that we could back our vm cluster with triple replicated ceph storage over LIO
[1:48] <noob2> i might be introducing a single point of failure though so i have to be careful with the proxy
[1:49] <iggy> should be able to double up the proxies and multipath pretty easily
[1:50] <noob2> iggy: yeah the multipathing is something i don't quite understand yet
[1:50] <noob2> LIO supports alua but how to use that i'm still uncertain
[1:50] <iggy> it's really not complicated if you understand linux multipathing first
[1:51] <noob2> i have homework to do then :)
[1:51] <iggy> which i guess i have a bit of a headstart on having worked for a storage company
[1:51] <noob2> yeah. i started learning how to use fibre about a year ago
[1:52] <noob2> i'm ok with the fabric and zoning but haven't really gotten into the bits and pieces of multipathing
[1:52] <iggy> new to the biz or just didn't do much storage before?
[1:53] <noob2> haven't done much storage before
[1:53] <noob2> i was a sysadmin at a college for a bunch of years before that. the unix admins didn't let me play with the san's
[1:53] <noob2> this new place i'm at lets me work with whatever
[1:53] <iggy> sounds fun
[1:53] <noob2> yeah i'm enjoying it
[1:53] <iggy> i like places like that
[1:54] <noob2> yeah they're open minded
[1:54] <noob2> hence why we're building a ceph cluster :D
[1:54] <noob2> iggy: so how would you setup a multipath'd proxy?
[1:54] <iggy> strange though to mention them spnding 10k/TB and ceph in the same convo
[1:54] <noob2> what do you mean?
[1:55] <iggy> 10k/TB people usually aren't open minded
[1:55] <noob2> exactly
[1:55] <noob2> they're usually hard headed, enterprise san only with support contracts
[1:55] <noob2> i donno. since i got there they seem to be changing their tune
[1:55] <noob2> i've implemented a lot of open source things that have impressed them
[1:56] <noob2> a year ago i would've gotten shot down on this hard core
[1:56] <iggy> 2 machines, 2 different IPs, lio on both, assign each host as a path, one goes down, multipath layer keeps going on the other
[1:57] <iggy> test till your eyes bleed
[1:57] <joao> sagewk, still around?
[1:57] <noob2> yeah i think i'm thinking the same thing now that i read what multipath does
[1:57] <noob2> both hosts mount the rbd and export over LIO as a path
[1:57] <noob2> yeah
[1:57] <iggy> correct
[1:57] <noob2> test till it hurts
[1:58] <iggy> the test till your eyes bleed part is important
[1:58] <noob2> haha
[1:58] <noob2> yeah i agree
[1:58] <noob2> weird issues that don't crop up until you're in prod would be awful
[1:58] <iggy> at that point you're stacking enough layers that you want to do a good bit of testing
[1:58] <noob2> it's possible i could pick up some caching benefit on the proxy also
[1:59] <noob2> right
[1:59] <iggy> maybe at the lio layer, the rbd kernel client doesn't cache yet
[1:59] <noob2> ok
[2:00] <noob2> LIO claims to have cache safe block io
[2:01] <iggy> i'm pretty sure the vmware multipath/block plugins are at the esx kernel level, but it would be cool to use librbd in vmware somehow
[2:01] <noob2> yeah that would be really cool
[2:01] <noob2> and save a lot of infrastructure build out
[2:01] <noob2> i think ovirt might be working on some patches to use ceph also
[2:02] <iggy> well, newer qemu's has librbd support built in
[2:02] <noob2> exactly
[2:02] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) Quit (Remote host closed the connection)
[2:02] <noob2> all they would need to do is build the other pieces to expose it
[2:02] <iggy> rh backports tons of stuff, so not surprising
[2:02] <noob2> yeah i know
[2:03] <iggy> i just left an all rh shop
[2:03] <noob2> oh yeah?
[2:03] <noob2> i'm at basically an all rh shop now
[2:03] <noob2> it's like 99%
[2:04] <iggy> trying to explain to my boss the dynamic for kvm was "frustrating"
[2:04] <noob2> haha
[2:04] <noob2> yeah i feel like redhat is holding me back in terms of adopting new technologies
[2:05] <noob2> it tends to cause problems which ubuntu doesn't have
[2:05] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:05] <iggy> +/-'s
[2:06] <noob2> yeah
[2:06] <noob2> stability vs new features
[2:06] <noob2> the place i'm at moves fast so i'm surprised they stabilized on rh
[2:07] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[2:08] <iggy> it's not bad if you're in the first year or two after major release
[2:08] <noob2> yeah
[2:09] <iggy> it's after that (before the next version) that life sucks all over
[2:09] <noob2> a bunch of stuff we have is redhat 5.4 because documentum or oracle insists on it
[2:09] <noob2> yeah that's a pain
[2:09] <iggy> yeah, i worked for an oil company, so it was all about the app vendors
[2:10] <noob2> i see
[2:10] <noob2> i'm sure they liked the stability of rh
[2:10] <iggy> the app vendors maybe...
[2:10] <noob2> haha
[2:11] <iggy> i don't think any of the admins at any of the offices liked having to try to get usb3 to work on rh5
[2:11] <iggy> it doesn't by the way
[2:11] <noob2> oh god
[2:11] <iggy> it's kind of iffy even on rh6
[2:12] <noob2> yeah i had this problem with fuse on < 5.4
[2:12] <noob2> what a pain
[2:15] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:17] * sagelap1 (~sage@90.sub-70-199-242.myvzw.com) has joined #ceph
[2:17] * sagelap (~sage@38.122.20.226) Quit (Ping timeout: 480 seconds)
[2:25] * sagelap1 (~sage@90.sub-70-199-242.myvzw.com) Quit (Ping timeout: 480 seconds)
[2:32] * MK_FG (~MK_FG@188.226.51.71) Quit (Quit: o//)
[2:35] * MK_FG (~MK_FG@188.226.51.71) has joined #ceph
[2:36] * noob2 (a5a00214@ircip4.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[2:36] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) has joined #ceph
[2:37] * sagelap (~sage@132.sub-70-197-146.myvzw.com) has joined #ceph
[2:37] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[2:49] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) Quit (Quit: Leaving.)
[2:51] * sagelap (~sage@132.sub-70-197-146.myvzw.com) Quit (Ping timeout: 480 seconds)
[3:11] * weber (~he@219.85.248.221) has joined #ceph
[3:13] * gucki (~smuxi@HSI-KBW-082-212-034-021.hsi.kabelbw.de) Quit (Ping timeout: 480 seconds)
[3:26] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) has joined #ceph
[3:30] * Cube (~Cube@173-119-9-75.pools.spcsdns.net) Quit ()
[3:33] * jlogan1 (~Thunderbi@72.5.59.176) Quit (Ping timeout: 480 seconds)
[3:34] * gucki (~smuxi@HSI-KBW-082-212-034-021.hsi.kabelbw.de) has joined #ceph
[3:49] * weber (~he@219.85.248.221) Quit (Remote host closed the connection)
[3:53] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has joined #ceph
[3:53] <plut0> how does ceph compare to lustre?
[4:01] <slang> sagelap: around?
[4:04] <slang> plut0: I would say the big difference is in fault-tolerance
[4:05] <slang> plut0: ceph was designed from the ground up to run on commodity storage hardware and provide replication, failure detection, rebuild of lost replicas, etc.
[4:05] <plut0> slang: ceph automatically replicates?
[4:05] <slang> plut0: yes
[4:05] <plut0> slang: and you can pick how many replicas?
[4:05] <slang> plut0: lustre is traditionally run on enterprise storage that has built in fault-tolerance
[4:06] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[4:06] <slang> plut0: yes with ceph you can choose the replication factor
[4:06] <plut0> slang: and how does ceph handle failures?
[4:07] <slang> plut0: it uses a heartbeat scheme to detect failures, and a set of paxos nodes (called monitors) to provide consistency and group membership
[4:09] <plut0> slang: so what does it do when a failed osd comes back online?
[4:09] <slang> plut0: ceph also has an object storage interface (called rados), and rest interface (called the rados gateway) that provides s3 functionality
[4:10] <slang> plut0: I'm not sure where lustre stands there
[4:11] <slang> plut0: it looks like lustre 2.0 has some kind of replication available
[4:11] <plut0> slang: does it stripe data across all osd's?
[4:12] <slang> plut0: no it uses something it calls placement groups to "shard" the data
[4:12] <plut0> slang: and how are these placement groups configured?
[4:12] <slang> plut0: there's good documentation on the architecture and details if you're interested
[4:14] <plut0> slang: i've been reading, trying to get a better understanding
[4:15] * adjohn (~adjohn@69.170.166.146) Quit (Quit: adjohn)
[4:16] <plut0> slang: you get to configure which osd's are in the pg's?
[4:18] <slang> plut0: not really, what you can control is what sets of osds can be grouped into a placement group
[4:19] <plut0> slang: i see
[4:19] <slang> plut0: for example, to avoid replicas on the same rack, the crush map allows you to specify that you want osds to be chosen for a placement group that don't include two of the osds in the same rack (or row, or whatever your failure domain is)
[4:20] <plut0> slang: how is the performance?
[4:24] * Cube (~Cube@12.248.40.138) has joined #ceph
[4:24] <plut0> slang: any benchmarks out there?
[4:31] <slang> plut0: this is the most recent one I know about: http://ceph.com/community/ceph-performance-part-1-disk-controller-write-throughput/
[4:32] <slang> plut0: performance is dependant on your hardware setup of course, but in my experience its quite good
[4:32] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[4:32] * Cube (~Cube@12.248.40.138) has joined #ceph
[4:34] <plut0> slang: well its scalable so it should scale as high as you design it right?
[4:35] <slang> plut0: ideally, yes :-)
[4:35] <slang> plut0: I would like to see a performance study with 10,000 or so osds
[4:36] <plut0> slang: yeah. i'm interested
[4:41] <wilson> i still struggle with journal device selection.
[4:44] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Quit: Leaving.)
[4:47] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[4:49] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has left #ceph
[4:49] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) has joined #ceph
[4:50] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit ()
[5:10] * mdrnstm (~mdrnstm@206-169-78-213.static.twtelecom.net) Quit (Quit: Leaving.)
[5:29] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[5:33] * yoshi (~yoshi@p14227-ipngn2501marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[6:17] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Quit: Leaving.)
[6:17] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[6:25] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Ping timeout: 480 seconds)
[6:56] * joshd1 (~jdurgin@2602:306:c5db:310:10b6:35a8:dee6:c0b) Quit (Quit: Leaving.)
[7:51] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[7:59] * gaveen (~gaveen@112.134.112.222) has joined #ceph
[8:04] * silversu_ (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[8:05] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[8:11] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Ping timeout: 480 seconds)
[8:12] * Solver (~robert@atlas.opentrend.net) Quit (Quit: Changing server)
[8:13] * sjustlaptop1 (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[8:17] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[8:19] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[8:37] * ctrl (~Nrg3tik@78.25.73.250) has joined #ceph
[8:39] * loicd (~loic@93.158.6.91) has joined #ceph
[8:42] * kees_ (~kees@devvers.tweaknet.net) has joined #ceph
[8:46] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[8:48] * loicd (~loic@93.158.6.91) Quit (Quit: Leaving.)
[8:57] * gregorg_taf (~Greg@78.155.152.6) has joined #ceph
[8:58] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[9:03] * loicd (~loic@193.49.201.35) has joined #ceph
[9:07] <fmarchand> Hi !
[9:07] <fmarchand> Is there a way to limit osd cache size ?
[9:07] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[9:09] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[9:09] * sabayonuser (~sabayonus@78.107.155.77) has joined #ceph
[9:10] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit ()
[9:28] * Leseb (~Leseb@193.172.124.196) has joined #ceph
[9:29] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:31] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:34] * fmarchand (~fmarchand@212.51.173.12) Quit (Quit: Leaving)
[9:39] <sabayonuser> Hello! Tell me, someone tried to mount CEPH storage on virtual host with mount.ceph tool?
[9:39] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:40] <sabayonuser> Virtual host running under libvirt/qemu
[9:41] * sabayonuser is now known as sin
[9:42] <sin> No one?
[9:46] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:48] * fmarchand (~fmarchand@212.51.173.12) has joined #ceph
[9:48] * fmarchand (~fmarchand@212.51.173.12) Quit ()
[9:51] * fmarchand (~fmarchand@212.51.173.12) has joined #ceph
[9:57] * yoshi (~yoshi@p14227-ipngn2501marunouchi.tokyo.ocn.ne.jp) Quit (Read error: Connection reset by peer)
[10:04] * yoshi (~yoshi@p15015-ipngn1501marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[10:10] * yoshi (~yoshi@p15015-ipngn1501marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[10:23] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:23] * yoshi (~yoshi@p15015-ipngn1501marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[10:24] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Remote host closed the connection)
[10:24] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[10:29] * miroslav (~miroslav@64.55.78.101) has joined #ceph
[10:29] * miroslav1 (~miroslav@64.55.78.101) has joined #ceph
[10:29] * miroslav (~miroslav@64.55.78.101) Quit (Read error: Connection reset by peer)
[10:34] * miroslav1 (~miroslav@64.55.78.101) Quit (Quit: Leaving.)
[10:44] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[11:02] * match (~mrichar1@pcw3047.see.ed.ac.uk) has joined #ceph
[11:06] * vagabon (~fbui@au213-1-82-235-205-153.fbx.proxad.net) has joined #ceph
[11:07] <vagabon> hi. Could anybody point me out some docs in order to evaluate/test ceph on a single host ?
[11:09] <fmarchand> Hi !
[11:09] <fmarchand> http://ceph.com/docs/master/start/quick-start/
[11:09] <fmarchand> this is a good start
[11:10] <vagabon> fmarchand: thanks !
[11:10] <fmarchand> you welcome
[11:10] <vagabon> I managed to miss that doc somehow
[11:11] <fmarchand> I'm a newbee too but I already did some stuff with ceph I'm still trying to understand a bit more
[11:11] <sin> Could anybody tell me why i can`t see all space of CEPH storage when mounting with mount.ceph tool?
[11:11] <fmarchand> where are you from ?
[11:12] <vagabon> fmarchand: france
[11:12] <fmarchand> sin : what do you mean ?
[11:12] <fmarchand> pareil
[11:12] <vagabon> fmarchand: your name sounds like your french too :)
[11:12] <vagabon> fmarchand: j'avais devine :)
[11:12] <fmarchand> juste un peu ...
[11:13] <vagabon> ?
[11:15] * yoshi (~yoshi@p15015-ipngn1501marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[11:15] <sin> i have 3Tb CEPH storage but when mounting i can see only 11Gb
[11:21] <fmarchand> when you do a ceph -s or ceph -w what do you have ?
[11:21] * gaveen (~gaveen@112.134.112.222) Quit (Ping timeout: 480 seconds)
[11:21] <fmarchand> vagabon : je veux dire que ca fait tres francais oui marchand :)
[11:22] <sin> ceph -s
[11:22] <sin> health HEALTH_OK
[11:22] <sin> monmap e1: 1 mons at {a=192.168.122.1:6789/0}, election epoch 1, quorum 0 a
[11:22] <sin> osdmap e11: 1 osds: 1 up, 1 in
[11:22] <sin> pgmap v119: 384 pgs: 384 active+clean; 21484 bytes data, 1060 MB used, 2760 GB / 2761 GB avail
[11:22] <sin> mdsmap e9: 1/1/1 up {0=a=up:active}
[11:23] <fmarchand> it looks ok ...
[11:23] <fmarchand> what command do you use to see the 11Gb thing ?
[11:23] <vagabon> fmarchand: ah ok
[11:24] <sin> df
[11:24] <vagabon> j'ai du mal en (aussi) francais :)
[11:24] <vagabon> j'ai du mal en (aussi) en francais :)
[11:24] <sin> 'df -h' to be correct
[11:25] <joao> if you guys keep up with the french, I'll start spitting out Portuguese literature :p
[11:27] <fmarchand> joao : no pb I love portugese !
[11:27] <fmarchand> :)
[11:28] <joao> well, this certainly backfired :p
[11:29] * gaveen (~gaveen@112.134.113.153) has joined #ceph
[11:29] <sin> What about Russian language?
[11:29] * gaveen (~gaveen@112.134.113.153) Quit ()
[11:31] <match> Мы не говорим России.
[11:32] * gaveen (~gaveen@112.134.113.153) has joined #ceph
[11:39] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[11:40] * gaveen (~gaveen@112.134.113.153) Quit (Quit: Leaving)
[11:41] * gaveen (~gaveen@112.134.113.153) has joined #ceph
[11:41] * maxiz (~pfliu@111.192.251.7) Quit (Ping timeout: 480 seconds)
[11:44] <ctrl> :)
[11:45] <ctrl> active just one osd
[11:47] <ctrl> how many osd`s in your ceph.conf?
[11:47] <iltisanni> ceph mds set_max_mds 1 -> and everything is OK
[11:48] <iltisanni> sorry wanted to post somewhere else
[11:48] * ScOut3R (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) has joined #ceph
[11:55] <ScOut3R> Hello dear ceph users! May i ask your opinion on two specific HBAs? I'm planning to build a small ceph cluster and the only question left is to which HBA should i buy because i don't have any previous experience with them.
[11:56] <sin> 'ceph mds set_max_mds 1' did not help
[11:59] <sin> <host>:/ 11G 4.2M 11G 1% /mnt
[12:00] <ctrl> hmm... hba for FC? IB?
[12:00] <ctrl> or ethernet?
[12:01] <sin> ethernet
[12:01] <sin> sory
[12:01] <ctrl> it`s ok )
[12:01] <ScOut3R> ctrl: just to access directly attached devices, so more like simple JBOS SAS controllers
[12:04] <ctrl> ScOut3R : Do you already choose something?
[12:05] <ScOut3R> ctrl: The two candidates are LSI's 9207-8i and Areca's 1320-8i card.
[12:06] <ctrl> sec, i will check )
[12:06] * loicd (~loic@193.49.201.35) Quit (Ping timeout: 480 seconds)
[12:08] <ScOut3R> for additional info: i'm planning to run 36 2TB drives with one controller (or with two using MPIO, it depends on the budget)
[12:10] <kees_> wasn't there a nice blog on ceph about hba's?
[12:11] <ScOut3R> kees_: there were two benchmark posts but it reviewd mostly raid controllers
[12:14] * silversu_ (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[12:14] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[12:16] <ctrl> ScOut3R: is lsi have external sas ports? can`t find about this. But areca have 2 external sff
[12:17] <ctrl> ScOut3R: yeah, i think areca will be better choice :) but a i don`t know about drivers in *nix
[12:17] <kees_> my dell perc 5's (rebranded lsi) have two external ports as well
[12:18] <ScOut3R> these two cards don't have external ports, but there are LSI and Areca cards that have; anyway, we won't need any external ports, if we want to expand the cluster i prefer buying more nodes to add more redundancy than connecting JBOD chassis to the already running nodes
[12:18] <kees_> lsi support is quite alright, although sometimes you struggle a bit with monitoring tools if you don't run redhat/suse
[12:18] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[12:19] <ScOut3R> i was brought to LSI because i used 3ware cards, but i guess LSI does not have that quality
[12:19] <fmarchand> I sent a mail to ceph-devel@vger.kernel.org .... I got a mail delivery error .... it's not the right email address for the mailing list ?
[12:20] <ctrl> fmarchand: yeah, right email. u should send not in html. just text
[12:21] <fmarchand> here is my pb anyway ... maybe somebody had the same thing : I have a weird pb. I remove a folder using a mounted fs partition. I did it and it worked well.
[12:21] <fmarchand> I checked later to see if I had all my folders in ceph fs ... : the folder I removed was back and I can't remove it ! Here is the error message I got :
[12:21] <fmarchand> rm -rf 2012-11-10/
[12:21] <fmarchand> rm cannot remove `2012-11-10': Directory not empty
[12:21] <fmarchand> This folder is empty ...
[12:22] <fmarchand> ctrl : thx ! I try with text ..
[12:26] <fmarchand> any idea ?
[12:26] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Ping timeout: 480 seconds)
[12:31] <kees_> http://pastie.org/private/b3jdsb0q4plodd6iyvjnq << does that mean virsh doesn't have rbd support? but why is it able to attach the disk?
[12:37] <ctrl> kees_: is qemu support rbd?
[12:38] <kees_> qemu-img does, and kvm does boot as well
[12:39] <kees_> and ubuntu installs just fine on the vm i created on an rbd disk
[12:40] <ctrl> ok? show "virsh -V"
[12:41] <ctrl> i think libvirt compile without rbd support
[12:41] <kees_> http://pastie.org/5371154
[12:41] <kees_> guess you're right
[12:42] <ctrl> kees_: yeah
[12:42] <kees_> hoped ubuntu 12.10 would support it more
[12:43] <ctrl> yep, me too
[12:43] <kees_> ah well, recompiling libvirt isn't that bad
[12:44] <ctrl> kees_: after compiling plz write here about your results, ok? :)
[12:45] <match> kees_:Different OS, but I had to recompile for RHEL6 for rbd support - worked pretty seamlessly
[12:45] <kees_> will do
[12:48] <ctrl> kees_: thx!
[12:50] <dweazle> match: did you do a manual compile or did you update the spec file? in the latter case: care to share? :)
[12:51] <dweazle> match: also, do you run osd's on rhel6 as well? and if so: have you patched glibc for syncfs() support?
[12:51] <match> dweazle: Just tweaked the existign spec-file. Only played a bit with it - not looked at syncfs at all
[12:51] <joao> can anyone check if the tracker works?
[12:51] <joao> seems down to me
[12:52] <dweazle> match: ah ok, i do think there's a serious interest in ceph on rhel (derative) stacks, and i don't expect redhat to start supporting it anytime soon
[12:52] <kees_> joao, tracker seems dead here as well
[12:53] <joao> kees_, thanks
[12:53] <match> dweazle: Yeah - our default env here is RHEL6 - but I'm beginning to think that any production setup with ceph will end up being ubuntu
[12:55] <dweazle> match: we're running everything on scientific linux (also rhel based), and on first glance it seems fairly trivial to fully support ceph on rhel
[12:55] <dweazle> match: i would applaud a community effort to make rhel a first class platform for ceph and help out wherever needed
[12:56] <match> dweazle: I agree - sl6 here too actually - and while it would make my life simpler to stick with sl, I'm also aware that it might just not happen.
[13:00] <joao> what would be needed to push ceph onto rhel or sl?
[13:00] <dweazle> match: i think it's just a matter of time :) if we're doing to do ceph we might consider setting up such a community ourselves anyway, no point in doing all the work and keeping it to ourselves, we don't have the resources to do everything ourselves
[13:01] <dweazle> joao: not that much i think, syncfs() is already in the kernel, glibc only needs a small patch, then you would need to compile rbd support into libvirt and qemu-kvm (might need some backporting work), that's about it
[13:01] <dweazle> that's aside from the ceph packages themselves, which are provided by the community already
[13:01] <joao> maybe you should inquire the list then
[13:02] <joao> I heard some chatter about pushing ceph onto rhel a while back, but have no idea how that ended up
[13:02] <joao> I thought it would be much more complex than that; I even thought some approval from rh would be needed
[13:02] <kees_> ah nice, it works after a recompile ctrl
[13:03] <joao> have no idea how those things work on enterprise editions ;)
[13:03] <kees_> all i had to do was install the librbd-dev and apt-get source libvirt --compile
[13:03] <ctrl> kees_: great! thx!
[13:04] <dweazle> joao: approval from rh is only needed when you want rh to include those bits by default, it's certainly allowed to patch/recompile rhel packages, but i don't think you are allowed to redistribute them if there is some rh branding in the packages, so you might be better off patching centos or sl packages then
[13:04] <joao> I see
[13:04] <dweazle> joao: perhaps i'm underestimating the amount of work needed, though
[13:06] <match> dweazle: I can at least confirm that it's easy enough to build ceph, libvirt and qemu with rbd suupport from the source and specfiles delivered from the respective 'vendors'.
[13:07] <dweazle> match: that's good to know :)
[13:09] <match> This kinda setup, all running on sl6.3: http://www.woodwose.net/thatremindsme/2012/10/ha-virtualisation-with-pacemaker-and-ceph/
[13:12] <dweazle> match: ah nice
[13:13] * loicd (~loic@90.84.144.56) has joined #ceph
[13:31] * loicd (~loic@90.84.144.56) Quit (Ping timeout: 480 seconds)
[13:38] * loicd (~loic@90.84.144.56) has joined #ceph
[14:00] * mdawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[14:12] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[14:20] * darkfader (~floh@188.40.175.2) has joined #ceph
[14:26] * darkfaded (~floh@188.40.175.2) Quit (Ping timeout: 480 seconds)
[14:29] * loicd1 (~loic@90.84.144.56) has joined #ceph
[14:29] * loicd (~loic@90.84.144.56) Quit (Quit: Leaving.)
[14:39] * loicd1 (~loic@90.84.144.56) Quit (Ping timeout: 480 seconds)
[14:45] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[14:49] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[14:54] <vagabon> fmarchand: I followed the "getting started" document and get at last :
[14:54] <vagabon> # ceph health
[14:54] <vagabon> HEALTH_WARN 576 pgs stuck inactive; 576 pgs stuck unclean
[14:54] * loicd (~loic@90.84.144.249) has joined #ceph
[14:54] <vagabon> do you know what's going wrong ?
[14:55] <fmarchand> mmm how many osd do you have ?
[14:56] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[14:56] <fmarchand> if you do ceph osd tree ?
[14:57] <vagabon> fmarchand: just a few secs please I'm reboot the host
[14:57] <vagabon> s/secs/mins :)
[14:57] <fmarchand> oki :)
[15:01] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:02] <vagabon> fmarchand: http://pastebin.com/prdSb4Xg
[15:02] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[15:02] * tryggvil_ is now known as tryggvil
[15:02] <fmarchand> and a ceph -s
[15:03] <vagabon> fmarchand: http://pastebin.com/rTfa12Wy
[15:03] <fmarchand> health HEALTH_OK
[15:03] <fmarchand> so ... no pb anymore :)
[15:03] <vagabon> :)
[15:04] <vagabon> the question is: "why I got this HEALTH_WARN at start ?"
[15:04] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[15:05] <fmarchand> your reboot ran service ceph start so ... it ran all osd/mds/mon ...
[15:05] <fmarchand> maybe time to warmup
[15:05] <fmarchand> you have 2osd's of 4To ?
[15:05] <vagabon> I don't think so.
[15:06] <vagabon> Actually I blindly followed the getting started doc.
[15:06] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[15:06] <vagabon> just to check that my ceph build was ok.
[15:06] <fmarchand> 2007 MB / 8969 MB avail with 2 osd's up (according to the ceph osd tree command)
[15:06] <vagabon> fmarchand: BTW ceph is not started at boot
[15:07] <fmarchand> So I assume ... there 4To each
[15:07] <fmarchand> how much memory ceph-osd take ?
[15:07] <tnt> count 1G per drive
[15:07] <vagabon> fmarchand: you probably meant 4Go right ?
[15:08] <fmarchand> sorry ... yes ... I just ate so my brain is not focused on reading ... :)
[15:09] <vagabon> ceph-osd is taking 887760KB
[15:09] <vagabon> (virtual mem size)
[15:11] <vagabon> I still don't understand why a "ceph -s" is needed in order to make "ceph health" happy.
[15:14] * maxiz (~pfliu@222.128.157.196) has joined #ceph
[15:15] * maxiz_ (~pfliu@222.128.157.196) has joined #ceph
[15:16] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Remote host closed the connection)
[15:16] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[15:17] * loicd (~loic@90.84.144.249) Quit (Ping timeout: 480 seconds)
[15:22] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) has joined #ceph
[15:26] * drokita (~drokita@199.255.228.10) has joined #ceph
[15:26] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[15:27] * drokita1 (~drokita@199.255.228.10) has joined #ceph
[15:27] * drokita1 (~drokita@199.255.228.10) Quit ()
[15:28] * drokita1 (~drokita@199.255.228.10) has joined #ceph
[15:30] * drokita1 (~drokita@199.255.228.10) Quit ()
[15:31] * drokita1 (~drokita@199.255.228.10) has joined #ceph
[15:34] * loicd (~loic@magenta.dachary.org) has joined #ceph
[15:34] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Read error: Connection reset by peer)
[15:34] * drokita (~drokita@199.255.228.10) Quit (Ping timeout: 480 seconds)
[15:35] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[15:35] * ChanServ sets mode +o scuttlemonkey
[15:36] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[15:37] * A85 (~user@p8183-ipbfp3701osakakita.osaka.ocn.ne.jp) has joined #ceph
[15:41] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) Quit (Remote host closed the connection)
[15:42] <Robe> omg omg omg jeff darcy is following me on twitter!
[15:43] <liiwi> .oO(what one click can do?)
[15:43] <Robe> hehehe
[15:43] <Robe> indeed
[15:53] * calebamiles (~caleb@c-24-128-194-192.hsd1.vt.comcast.net) has joined #ceph
[16:00] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) has joined #ceph
[16:02] <kees_> anyone familiar with phprados and this error? http://pastie.org/5371959
[16:04] <tnt> I guess it's mismatch between the version of ceph it expects and what you have.
[16:08] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) has joined #ceph
[16:18] * yanzheng (~zhyan@jfdmzpr05-ext.jf.intel.com) Quit (Remote host closed the connection)
[16:28] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[16:28] * loicd (~loic@magenta.dachary.org) has joined #ceph
[16:29] * yanzheng (~zhyan@134.134.137.73) has joined #ceph
[16:33] * kees_ (~kees@devvers.tweaknet.net) Quit (Remote host closed the connection)
[16:40] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[16:41] * yanzheng (~zhyan@134.134.137.73) Quit (Quit: Leaving)
[16:41] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit ()
[16:47] <jtang> no one famous follows me on twitter
[16:47] <jtang> cause im a nobody!
[16:56] * calebamiles (~caleb@c-24-128-194-192.hsd1.vt.comcast.net) Quit (Remote host closed the connection)
[17:04] * sin (~sabayonus@78.107.155.77) Quit (Read error: Connection reset by peer)
[17:06] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[17:06] * vagabon (~fbui@au213-1-82-235-205-153.fbx.proxad.net) has left #ceph
[17:06] * jlogan1 (~Thunderbi@2600:c00:3010:1:fc51:3370:98cf:1d11) has joined #ceph
[17:10] * calebamiles (~caleb@c-24-128-194-192.hsd1.vt.comcast.net) has joined #ceph
[17:10] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:11] * James259 (~James259@94.199.25.228) has joined #ceph
[17:12] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) Quit (Quit: Ex-Chat)
[17:13] <James259> 2012-11-13 16:12:49.219048 osd.0 [WRN] client.12665 10.0.0.1:0/2144420526 misdirected client.12665.1:2182408 2.34 to osd.0 not [1,0,3] in e2012/2012
[17:13] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[17:13] <James259> ^^ is this bad?
[17:13] * loicd (~loic@magenta.dachary.org) has joined #ceph
[17:14] <James259> OSD.0 was put 'out' a few weeks ago and put back 'in' today.
[17:14] <James259> replication level is 2 too.. not sure if [1,0,3] is correct. Should I be worried?
[17:15] * timmclaughlin (~timmclaug@69.170.148.179) has joined #ceph
[17:16] * BManojlovic (~steki@91.195.39.5) Quit (Remote host closed the connection)
[17:19] * Leseb (~Leseb@193.172.124.196) Quit (Quit: Leseb)
[17:29] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[17:30] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[17:33] * mdawson (~chatzilla@166.137.98.175) has joined #ceph
[17:36] <James259> oh.. version might be helpful... ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
[17:36] * mdawson (~chatzilla@166.137.98.175) Quit (Read error: Connection reset by peer)
[17:39] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:39] * ScOut3R (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) Quit (Remote host closed the connection)
[17:39] * sagelap (~sage@76.89.177.113) has joined #ceph
[17:45] <James259> gotta go afk for a while. If anyone see's this and does have suggestions - please leave them for me and I will see them when I get back.
[17:54] * aliguori (~anthony@32.97.110.59) has joined #ceph
[18:02] * match (~mrichar1@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[18:03] * match (~mrichar1@pcw3047.see.ed.ac.uk) has joined #ceph
[18:05] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[18:06] * sagelap1 (~sage@161.sub-70-197-152.myvzw.com) has joined #ceph
[18:07] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[18:07] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[18:08] * drokita1 (~drokita@199.255.228.10) has left #ceph
[18:08] * sagelap (~sage@76.89.177.113) Quit (Ping timeout: 480 seconds)
[18:17] * sjustlaptop1 (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[18:23] <yehudasa> tnt: you asked a question about multipart upload into exiting key
[18:23] <yehudasa> that's a known issue, I pushed a fix for that not too long ago, which version are you running?
[18:29] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) has joined #ceph
[18:29] * sagelap1 (~sage@161.sub-70-197-152.myvzw.com) Quit (Read error: Connection reset by peer)
[18:32] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[18:32] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:33] * loicd (~loic@jem75-2-82-233-234-24.fbx.proxad.net) has joined #ceph
[18:34] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) has joined #ceph
[18:42] * miroslav1 (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[18:42] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Read error: Connection reset by peer)
[18:48] * sagelap1 (~sage@38.122.20.226) has joined #ceph
[18:52] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[18:52] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit ()
[18:56] * sagelap1 (~sage@38.122.20.226) Quit (Ping timeout: 480 seconds)
[18:58] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[18:59] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[19:03] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[19:03] * miroslav1 (~miroslav@wlan-clients-8094.sc12.org) Quit (Read error: Connection reset by peer)
[19:11] * sagelap (~sage@2607:f298:a:607:6181:8d89:542:7b42) has joined #ceph
[19:21] * A85 (~user@p8183-ipbfp3701osakakita.osaka.ocn.ne.jp) Quit (Quit: さようなら)
[19:34] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[19:35] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (Quit: Leaving.)
[19:39] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:39] * aliguori (~anthony@32.97.110.59) Quit (Remote host closed the connection)
[19:41] * aliguori (~anthony@32.97.110.59) has joined #ceph
[19:42] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[19:44] * houkouonchi-work (~linux@12.248.40.138) Quit (Read error: Connection reset by peer)
[19:45] * houkouonchi-work (~linux@12.248.40.138) has joined #ceph
[19:47] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[19:48] * loicd (~loic@jem75-2-82-233-234-24.fbx.proxad.net) Quit (Quit: Leaving.)
[19:50] * adjohn (~adjohn@69.170.166.146) has joined #ceph
[19:55] * glowell (~glowell@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[19:58] * danieagle (~Daniel@177.99.134.180) has joined #ceph
[20:11] * dmick (~dmick@2607:f298:a:607:1a03:73ff:fedd:c856) has joined #ceph
[20:16] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:18] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[20:18] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[20:18] * BManojlovic (~steki@85.222.180.134) has joined #ceph
[20:23] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[20:24] <James259> is this bad?
[20:24] <James259> 2012-11-13 16:12:49.219048 osd.0 [WRN] client.12665 10.0.0.1:0/2144420526 misdirected client.12665.1:2182408 2.34 to osd.0 not [1,0,3] in e2012/2012
[20:24] <James259> pages and pages of it scrolling by
[20:24] <James259> OSD.0 was put 'out' a few weeks ago and put back 'in' today.
[20:24] <James259> replication level is 2 too.. not sure if [1,0,3] is correct.
[20:24] <James259> Should I be worried?
[20:24] <James259> ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
[20:26] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:26] * ChanServ sets mode +o dmick
[20:26] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:27] <joshd> James259: are you using a kernel client (perhaps an older one) with anything special about your crushmap (like different crush tunables?)
[20:28] <James259> rbd client.
[20:28] <James259> not sure about crushmap. weights set based on capacity of each server. (Weight = Tb)
[20:28] <James259> is that what you mean by tunables?
[20:29] <joshd> no, I mean http://ceph.com/docs/master/cluster-ops/crush-map/#tunables
[20:29] <joshd> are you using 'rbd map'?
[20:31] <James259> sorry.. reading that doc you linked. (never seen that before so I probably have not tampered)
[20:31] * LarsFronius (~LarsFroni@95-91-242-155-dynip.superkabel.de) has joined #ceph
[20:32] <James259> rbd create --size=xx <img name>
[20:32] <James259> echo "mons name=admin rbd <img name> >/sys/bus/rbd/add
[20:32] <James259> thats pretty much all I did if that helps.
[20:32] <James259> not sure what rbd map is
[20:33] <James259> still reading that doc. :)
[20:33] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Quit: Leaving.)
[20:33] * BManojlovic (~steki@85.222.180.134) Quit (Quit: Ja odoh a vi sta 'ocete...)
[20:34] <benpol> So I've got a small cluster, 3 mons, 1 mds, 3 osd servers. I'm not doing anything with cephfs and was thinking it might be nice to remove the single mds from the picture. Can I simply remove the mds from the cluster's ceph.conf and restart ceph services on all nodes?
[20:34] * LarsFronius (~LarsFroni@95-91-242-155-dynip.superkabel.de) Quit ()
[20:34] <James259> I am running Kernel 3.2 if that means anything. (Noticed a mention of 3.5 in that doc)
[20:34] * LarsFronius (~LarsFroni@2a02:8108:380:12:49be:4e24:3b0:594a) has joined #ceph
[20:35] <joshd> 'rbd map' basically does that echo for you
[20:36] * BManojlovic (~steki@85.222.180.134) has joined #ceph
[20:36] <James259> ahh, thats good to know.
[20:36] <James259> I thnk your documentation has had an overhaul since I set this up.
[20:36] <joshd> could you pastebin your crushmap (i.e. ceph osd getcrushmap -o /tmp/cm && crushtool -d /tmp/cm)
[20:36] <joshd> yeah, the docs are a lot better and more comprehensive now
[20:37] <James259> Okay, pretty sure I am running default options for tunables. I have not run any such commands.
[20:37] <joshd> just don't trust the wiki
[20:37] <James259> lol
[20:37] <James259> I have always been very confused by the crushmap.. racks, hosts, etc.
[20:39] <James259> http://pastebin.com/U0uLrj3G
[20:40] * mdrnstm (~mdrnstm@206-169-78-213.static.twtelecom.net) has joined #ceph
[20:40] <James259> never seen that output before. :/ Bet I have something well messed up in there.
[20:43] <joshd> the general idea is that you can represent your failure domains as a tree e.g. these osds on these hosts in these racks use these switches
[20:43] <joshd> so you can have replicas placed in different failure domains, to limit the damage from correlated failures
[20:43] * Cube (~Cube@12.248.40.138) has joined #ceph
[20:44] <joshd> I suspect the issue is 'item unknownrack weight 0.000' in the 'pool default' section
[20:44] <James259> Yeah.. that is there from default set-up and I could not get rid of it.
[20:44] <James259> Set-up is scripted and could not find a command that would remove it.
[20:45] <joshd> you can edit the crushmap as text, then compile and re-inject it
[20:46] <joshd> just to see if that's the problem
[20:46] <joshd> if you're going to be using the kernel client, 3.6 is recommended
[20:47] <joshd> if you're using qemu/kvm the kernel client is unnecessary
[20:47] * vata (~vata@208.88.110.46) Quit (Remote host closed the connection)
[20:47] <James259> I am using bcache on top of RBD and that is only available for 3.2 at the moment - otherwise I would be running the latest stable kernel. :(
[20:47] * Cube1 (~Cube@12.248.40.138) has joined #ceph
[20:47] <James259> I use the kernel client so that I can run bcache on top of it.
[20:48] <joshd> ah, I see
[20:48] <joshd> how's bcache working out?
[20:48] <James259> very impressive.
[20:49] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[20:49] <James259> performance improvement was pretty incredible. Especially since the disks in the OSD's are WD Greens - so not so fast.
[20:49] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:50] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:50] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) has joined #ceph
[20:50] <joshd> nice
[20:51] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[20:52] <joshd> so does 'ceph osd crush remove unknownrack' fail?
[20:53] <James259> I will try it. It is more likely I was doing it wrong.
[20:53] <James259> root@Control:~/tmp# ceph osd crush remove unknownrack
[20:53] <James259> device 'unknownrack' does not appear in the crush map
[20:54] <joshd> ah, I guess that only works on devices (i.e. osds)
[20:54] * vata (~vata@208.88.110.46) has joined #ceph
[20:54] <James259> Yeah. I think I had the same problem trying to remove a host.
[20:57] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[20:57] <James259> following is last command I run. Do i need to do any more or just watch ceph -w?
[20:57] <James259> root@Control:~/tmp# ceph osd setcrushmap -i crushmap.out
[20:57] <James259> 2012-11-13 19:56:52.313610 7ffd2fa0f780 -1 read 602 bytes from crushmap.out
[20:57] <James259> set crush map
[20:58] <James259> I purely removed the unkonwnrack line.
[20:59] * gaveen (~gaveen@112.134.113.153) Quit (Remote host closed the connection)
[21:00] <James259> ceph -w is still scrolling pages of the same warnings but maybe it needs a little while to sort itself out. :)
[21:02] <benpol> found some mailing list posts about removing mds from a cluster... updated ceph.conf restarted ceph services on all nodes. but when I do a "ceph mds rm mds.a" I get back this response: "unable to parse positive integer 'mds.a'". I'm running 0.53, has the syntax for this command changed?
[21:04] <joshd> benpol: I think it's looking for the mds logical id rather than the daemon id
[21:09] <jtang> *cool* i got me a ceph tshirt :)
[21:09] <dmick> jtang: +1
[21:09] <James259> ooh, where?
[21:09] <benpol> joshd: ok, so.... how do I figure out the logical id of the mds :-D
[21:10] <joshd> benpol: I think it's in ceph -s, or ceph mds dump
[21:10] <jtang> James259: at supercomputing 2012
[21:11] <jtang> just meet rweeks and miroslav here
[21:11] <jtang> i might drop by to them again to have another chat, i didnt quite get through my list
[21:11] <jtang> of things to ask
[21:11] <James259> wow.. cool
[21:12] <jtang> i wish there were a few engineers here :P
[21:12] <jtang> at the conf that is
[21:12] <jtang> that would have been nice
[21:16] <benpol> joshd: it was in "ceph mds dump", apparently mine is 8365. but when I do "ceph mds rm 8365" it says "cannot remove active mds.a rank 0"
[21:17] * MoroIta (~MoroIta@host55-245-dynamic.16-87-r.retail.telecomitalia.it) has joined #ceph
[21:17] <joshd> benpol: I think you may have to stop it first
[21:18] <benpol> joshd: already did that (well before I tried to remove it)
[21:18] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has left #ceph
[21:19] * MoroIta (~MoroIta@host55-245-dynamic.16-87-r.retail.telecomitalia.it) Quit ()
[21:21] <joshd> benpol: did you do the newfs command already?
[21:22] <benpol> joshd: nope, trying it now
[21:22] <dmick> so vstart -X still doesn't work eh...
[21:22] <benpol> joshd: but "ceph mds newfs metadata data" returns: "ceph mds newfs metadata data"
[21:23] <benpol> heh
[21:23] <benpol> joshd: returns: "unable to parse positive integer 'metadata'"
[21:25] <joshd> that sounds like it's looking for pool ids too ('ceph osd dump' shows them)
[21:25] <dmick> 3- create a new mdsmap with those pools ceph mds newfs METADATAPOOLID DATAPOOLID
[21:25] <dmick> (from mail from Sage)
[21:28] <benpol> dmick: thanks for that
[21:29] <James259> joshd: Is the [2, 0, 4] part of the warnings something that is not right? I have it set to keep 2 copies (default I believe), not 3.. or is that just what it does when its moving data back to osd.0?
[21:29] <benpol> based on that I'm trying "ceph mds newfs 1 0", which returns an exhilarating warning: "this is DANGEROUS and will wipe out the mdsmap's fs, and may clobber data in the new pools you specify. add --yes-i-really-mean-it if you do."
[21:31] <benpol> Seems to have done the trick, thanks joshd and dmick!
[21:31] <joshd> James259: it may temporarily do that during recovery I think... could you pastebin 'ceph -s', 'ceph osd dump' and 'ceph pg dump'?
[21:33] * benpol wanders off
[21:33] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) Quit (Remote host closed the connection)
[21:33] <joshd> James259: are there any warnings in dmesg or syslog on your client machine from libceph or rbd?
[21:35] <gucki> is there any way to read/write data at a specific offset of an rbd image using the command line tools?
[21:37] <joshd> gucki: not easily from the cli, kernel map & dd or the python librbd bindings are pretty easy to do though
[21:38] <joshd> there's also qemu-io, although I don't remember its syntax
[21:38] <James259> joshd: first 2 - http://pastebin.com/qmfKt91Y
[21:39] <gucki> joshd: great thanks, i'll look into these :). just wanted to make sure there's no easy option i overlooked :)
[21:40] <James259> joshd: 3rd one.. http://pastebin.com/bpy5itV1
[21:40] <buck> I have a 1-line patch for autobuilder-ceph/fabfile.py that could use a review if anyone as a minute. Corrects the username to use when deploying to the squeeze gitbuilder host. Branch wip-hadoop
[21:40] <joshd> James259: ah, I think there was a kernel bug with handling remapped pgs that were then remapped again
[21:40] <James259> oh.
[21:41] <James259> 3.2.0-32
[21:41] <James259> oh, actually.. think that could be wrong kernel version.. one sec.
[21:42] <James259> sorry.. correct version (used on clients and osd's) is 3.2.28
[21:43] <James259> kern.log:-
[21:44] <James259> Nov 13 20:43:31 Node-1 kernel: [490977.634113] end_request: I/O error, dev rbd30, sector 988184
[21:44] <James259> Nov 13 20:43:31 Node-1 kernel: [490977.634119] end_request: I/O error, dev rbd30, sector 988200
[21:44] <James259> Nov 13 20:43:31 Node-1 kernel: [490977.636217] end_request: I/O error, dev rbd30, sector 988248
[21:44] <James259> Nov 13 20:43:31 Node-1 kernel: [490977.636244] end_request: I/O error, dev rbd16, sector 42638784
[21:44] <James259> pages and pages.
[21:46] <James259> pretty much the same in dmesg and syslog
[21:46] <joshd> James259: yeah, that looks like this bug: http://tracker.newdream.net/issues/2446
[21:47] <joshd> the fix is in 3.5
[21:47] <joshd> and if you're compiling your own kernel, it's a pretty small patch
[21:48] <James259> bcache is a full kernel tree and is fixed to 3.2.28 :(
[21:48] <James259> I have no option to go higher yet.
[21:49] <James259> Can I apply the patch to 3.2 and recompile?
[21:50] <joshd> yeah, it's a ceph-specific patch
[21:50] <joshd> it should apply fine
[21:50] <joshd> only needed on your client machines
[21:51] <James259> ahh, super. is it just the few lines Sage posted on the bug you linked?
[21:51] <joshd> yeah
[21:52] <James259> Ok, I will go try and figure that out. Looks pretty straight forward. One final question if you dont mind - is my data damaged? (I got about 100 live customers on here :/ )
[21:53] <James259> well.. at least would you expect it is damaged.
[21:54] <joshd> no, data isn't affected by this, just availability
[21:54] <James259> phew.. thank you so much for your help.
[21:55] <joshd> np
[21:55] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[21:59] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) has joined #ceph
[22:16] * jmcdice (~root@135.13.255.151) Quit (Ping timeout: 480 seconds)
[22:16] * joey_alu_ (~root@135.13.255.151) Quit (Ping timeout: 480 seconds)
[22:21] * scalability-junk (~stp@188-193-202-99-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[22:27] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:39] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[22:46] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[22:56] <dmick> anyone want to offer an opinion on https://github.com/dmick/ceph/commit/a08471dac57296672f79b8c236f4f8b0143f2ca9
[22:57] <dmick> quick "make vstart.sh -X work" review, not worth a branch
[23:01] <sagewk> dmick: looks good
[23:08] <slang> anyone else seeing osds hanging right away on a fresh deploy with master?
[23:09] * miroslav1 (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[23:09] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Read error: Connection reset by peer)
[23:12] <benpol> so when watching a cluster recover from an osd failure, I sometimes see some PGs stuck in a "stale+..." state. What exactly does "stale" mean in this context? One consequence of a bunch of PGs in a stale state seems to be that IO to rbd volumes stalls, is that expected?
[23:13] <joao> hey guys, when was 'scratchtool' added to the tree?
[23:15] <joao> nevermind
[23:15] <joao> git blame is my friend
[23:17] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[23:18] <sjust> slang: you mean from vstart/
[23:18] <sjust> ?
[23:18] <slang> sjust: from a simple teuthology run
[23:18] <sjust> hang on
[23:19] <slang> roles:
[23:19] <slang> - [mon.0, mon.1, mon.2, mds.0, osd.0, osd.1, osd.2, client.0]
[23:19] <slang> tasks:
[23:19] <slang> - ceph:
[23:19] <slang> - ceph-fuse: [client.0]
[23:19] <slang> - interactive: true
[23:19] <slang> woah
[23:19] <slang> nevermind
[23:19] <sjust> k
[23:20] <slang> I guess I'm just not patient enough
[23:33] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[23:34] * mdawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[23:40] * sagelap (~sage@2607:f298:a:607:6181:8d89:542:7b42) Quit (Ping timeout: 480 seconds)
[23:42] * sagelap (~sage@38.122.20.226) has joined #ceph
[23:44] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) Quit (Remote host closed the connection)
[23:51] * sagelap (~sage@38.122.20.226) Quit (Quit: Leaving.)
[23:52] * sagelap (~sage@38.122.20.226) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.