#ceph IRC Log

Index

IRC Log for 2013-01-12

Timestamps are in GMT/BST.

[0:00] <noob2> a bunch of write_same w/o unmap bit not supported
[0:00] <noob2> i think that is lio's error message
[0:01] <fghaas> gregaf, sjust, it turns out to be a conflation of a performance issue, a lying disk, and a bug.
[0:03] <gregaf> noob2: hmm, that doesn't mean much to me
[0:03] <gregaf> is it possible that the RBD device just got very latent and LIO couldn't handle it?
[0:03] <sjust> jmlowe: backtrace?
[0:03] <noob2> it's possible
[0:03] <sjust> fghaas: what happened?
[0:03] <gregaf> so the crash is not in the rbd code?
[0:03] <fghaas> issue #1. box with 14 osd used two ssds for journals, one of those ssds was also used for the OS install, mondir was not on a separate device. as gregaf pointed out, lots of I/O to all the journals meant that the mon was getting too slow I/O, and stopped responding in time.
[0:04] <noob2> i don't think so at this point. i saw some google pages saying that vmware 5 and LIO utils have an issue with latency
[0:04] <gregaf> noob2: and how did you actually change your rules?
[0:04] <noob2> i changed it from host to rack
[0:04] <noob2> i have 3 racks
[0:04] <noob2> after that i told the pool to use the new crush rule
[0:04] <gregaf> but nothing else?
[0:04] <noob2> nothing else
[0:04] <gregaf> I just want to establish it isn't use of the tunables or anything
[0:04] <noob2> ok
[0:04] <gregaf> okay
[0:04] <noob2> interesting
[0:05] <noob2> the gateway seems to continue working as long as i don't run dstat
[0:05] <noob2> as soon as i run that tool it blows up
[0:05] <fghaas> which exposes a deeper problem, really, which is that beyond a certain number of osds an ssd is in fact a pretty bad idea for journals, and you're much better off putting the journals on the spinners. and up to this point it's not really documented where that threshold is.
[0:05] <fghaas> so we put the journals on the spinners, and the dying mon problem went away.
[0:05] <noob2> nvm, it just crashed again
[0:05] <gregaf> fghaas: yeah, it depends on how fast the SSD is, if you're using xfs how many OSDs you can tolerate losing, etc
[0:06] <noob2> gah. i'm going to head home and continue this later :(
[0:06] <fghaas> issue #2. PGs stuck in peering. that was in fact a lemon of a disk (as sjust suggested). once we kicked that out, the peering problem disappeared
[0:06] <gregaf> noob2: well it ought to continue responding, but you're moving all your data so I wouldn't be surprised if latency got higher and unpredictable, and if the stack above it can't handle that I'm not sure what we can do… :(
[0:06] <noob2> yeah
[0:06] <noob2> this is bad. that means in the future i can't reboot a node without lio causing a kernel panic
[0:07] <fghaas> interestingly, that disk lies like there's no tomorrow about its own performance. writing to a 7.2krpm SATA drive in O_DIRECT mode with dd really ought not to return 150MB/s for a 2G write
[0:08] <gregaf> noob2: hmmm; yeah, in unexpected failures you're going to have latency while the system decides it's dead
[0:08] <noob2> yeah
[0:08] <sjust> fghaas: wow, heh
[0:08] <noob2> i guess i need to revisit my concept of using ceph over fibre
[0:08] <gregaf> for planned downtime you can just mark it down (but not out) and then reboot though, so a different OSD takes over right away
[0:09] <fghaas> once that was kicked out, it was smooth sailing until we tried a rados bench from a client node, which caused a mon to hit an assert and kill itself
[0:09] <noob2> ok
[0:09] <fghaas> 2013-01-11 23:56:42.979216 7f75a7020700 -1 mon/MonitorStore.cc: In function 'void MonitorStore::write_bl_ss(ceph::bufferlist&, const char*, const char*, bool)' thread 7f75a7020700 time 2013-01-11 23:56:42.977299
[0:09] <fghaas> mon/MonitorStore.cc: 382: FAILED assert(!err)
[0:09] <fghaas> which, in this case, didn't bring the cluster down (which is better than before :) ), but not exactly ideal either.
[0:09] <noob2> alright talk to you guys monday :)
[0:09] <gregaf> fghaas: well, that means the filesystem returned an error (probably on fsync)
[0:09] <noob2> have a nice weekend
[0:10] * noob2 (~noob2@ext.cscinfo.com) Quit (Quit: Leaving.)
[0:10] <gregaf> you too
[0:10] * gaveen (~gaveen@112.135.156.163) Quit (Ping timeout: 480 seconds)
[0:10] <gregaf> fghaas: oh, nope, on a writev, not an fsync
[0:11] <gregaf> in any case, appropriate behavior, although I guess it could be a little more polite about shutting down — it attempted to write data which needs to be durable to disk, and the filesystem said "no can do"
[0:11] <fghaas> gregaf: um what I see in dmesg at the time is a call trace from a hung task (which I set to a relatively low 10 secs to troubleshoot the other I/O problems), but I thought that that didn't exactly mean the kernel actually flagged an I/O error, or does it?
[0:13] <fghaas> at any rate, if that's actually ritual suicide in shame over an I/O error, fair enough
[0:13] <gregaf> I have no idea…but that assert(!err) is looking at the (very slightly obfuscated) errno from a syscall to writev
[0:13] <fghaas> alright, let me reset the hung_task_timeout to the regular 120s and see if I can reproduce the problem
[0:14] <fghaas> whoa!
[0:14] <fghaas> can't restart that mon now
[0:14] <fghaas> ah. nvm
[0:15] <fghaas> suppose that guy doesn't really like a full /var. presume I should have reverted sjust's earlier log level suggestions :)
[0:15] <gregaf> haha, yes
[0:15] <loicd> listening to this story is actually better than watching big bang theory
[0:17] <fghaas> leave it to the french guy to inject the sarcasm :)
[0:18] <gregaf> before I'd seen ~6 episodes of Big Bang Theory with other people I'd have agreed without any sarcasm ;)
[0:18] <loicd> fghaas: :-) no sarcasm at all, entertaining and educational
[0:18] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:18] <nhm> fghaas: hah! I just hit that error an hour ago during *my* customer testing. :)
[0:18] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:19] <nhm> fghaas: had to put the monitor data in /tmp!
[0:19] * gaveen (~gaveen@112.135.143.71) has joined #ceph
[0:21] <gregaf> Kioob: I had another look at that log and I wasn't interpreting it quite right (scrub messages don't show up the way I thought they did)
[0:22] <gregaf> but one of the things it definitely is showing is an OSD doing "push"es, which means it's sending another OSD objects that it should be storing but isn't
[0:22] <gregaf> are you sure the only output when your cluster gets slow is scrubbing? nothing about backfill or anything else?
[0:22] <gregaf> next time it does happen can you run "ceph -s" and pastebin the output?
[0:25] <fghaas> gregaf: yeah, giving the mon some room to write to helps :)
[0:25] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[0:27] <fghaas> we'll still have to figure out a few recommendations for the osd recovery tunables for setups with >=8 osds per node... if I kick out 10 at a time, rados bench performance drops massively, if I kick out 5, the impact is almost unnoticeable
[0:28] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:29] * loicd (~loic@magenta.dachary.org) has joined #ceph
[0:30] <fghaas> but at any rate, thanks for pointing us in the right direction, gregaf & sjust
[0:30] <fghaas> and nhm
[0:31] <gregaf> yep, setting those tunables correctly is going to be very important for real clusters of any size
[0:32] * danieagle (~Daniel@177.97.248.250) has joined #ceph
[0:32] * sjustlaptop (~sam@2607:f298:a:607:a951:43ad:db06:6330) has joined #ceph
[0:35] <Kioob> gregaf : yes, I'm sure sure sure. There is only scrubbing, I haven't got any backfill, recovering, etc
[0:36] * fghaas (~florian@91-119-215-212.dynamic.xdsl-line.inode.at) has left #ceph
[0:37] <Kioob> by disabling the scrubbing, the bandwith drop from 2Gbps to 10Mbps ! There was just 4 scrubs running
[0:38] * madkiss (~madkiss@p57A1CFCD.dip.t-dialin.net) Quit (Quit: Leaving.)
[0:38] <gregaf> can you enable it again, check that the bandwidth usage goes up as before, and then grab the output of ceph -s for me?
[0:39] <Kioob> I suppose I can throw a manual scrub ?
[0:39] <gregaf> if that really does match up then I believe you've run into a significant bug nobody else has seen and we'll want to do some more extravagant log collection and analysis
[0:39] <gregaf> hrmm, I don't remember if that will override the automatic settings or not, but I think it will, yes
[0:40] <Kioob> it works
[0:40] <Kioob> I had 5Mbps of bandwith. 3728pg �active+clean�.
[0:41] <Kioob> I start ceph pg scrub 3.64
[0:41] <Kioob> it jump to 1Gbps
[0:41] <Kioob> but.... it stop after about 10 seconds
[0:41] <Kioob> and scrub is finished
[0:41] <Kioob> last time, that PG took 40 minutes to do scrub
[0:42] <gregaf> hmm, sjust, can you see a scrub finishing that quickly? maybe if all the data was already cached?
[0:42] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[0:43] <Kioob> mm yes I have 48GB of RAM on each host
[0:43] <Kioob> and that PG have a size of 20GB
[0:45] <Kioob> is it possible that the scrub process go in a sort of infinite loop ?
[0:46] <jmlowe> sjust: http://pastebin.com/FbFSqaEN
[0:48] <gregaf> Kioob: how many OSDs on each host, again?
[0:48] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[0:48] <Kioob> 8
[0:49] <gregaf> based on what I'm seeing here, I really do think you just missed that some of the PGs were in some sort of recovery, and maybe your scrub is going really fast because you've got a bunch of cache relative to total data set size
[0:49] <Kioob> 8 OSD of 1To per host (and 48GB of RAM)
[0:49] * sjustlaptop (~sam@2607:f298:a:607:a951:43ad:db06:6330) Quit (Quit: Leaving.)
[0:50] <gregaf> the amount you actually have stored is the important one, not the capacity ;)
[0:50] * sjustlaptop (~sam@2607:f298:a:607:74b2:b825:7d2a:e817) has joined #ceph
[0:50] <Kioob> �ceph -w� report that, no ?
[0:50] <Kioob> pgmap v2346181: 3728 pgs: 3728 active+clean; 3654 GB data, 10143 GB used, 19504 GB / 29648 GB avail
[0:51] <Kioob> and recovery operations are in logs. And there is no recovery in logs
[0:51] <gregaf> yeah, so you've got 3.6TB of data and 200GB of cache ;)
[0:51] <Kioob> yes
[0:52] <gregaf> you're looking through the monitor log?
[0:52] <Kioob> and osd logs, and I was using �watch ceph status�
[0:53] * jpieper (~josh@209-6-86-62.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) has joined #ceph
[0:53] <gregaf> if you post your monitor logs somewhere I'll be interested in glancing through them
[0:55] <Kioob> ok I compress that
[0:56] * sjustlaptop (~sam@2607:f298:a:607:74b2:b825:7d2a:e817) Quit (Quit: Leaving.)
[0:57] <Kioob> there was the problem at � 23:45:00 �, Paris time, like my logs
[0:58] <Kioob> (not exactly 23h45, but near that)
[0:58] <gregaf> the problem? you mean the scrub and high network usage?
[0:58] <Kioob> and all production down because of huge latencies
[0:59] <Kioob> then I disable scrubbing in ceph.conf, and restart all OSD host per host
[0:59] <Kioob> then it was ok
[0:59] <Kioob> so logs : https://daevel.fr/copy.mon.a.log.gz
[0:59] <Kioob> (140MB)
[1:00] <gregaf> btw:
[1:00] <gregaf> wget https://daevel.fr/copy.mon.a.log.gz
[1:00] <gregaf> --2013-01-11 15:21:31-- https://daevel.fr/copy.mon.a.log.gz
[1:00] <gregaf> Resolving daevel.fr... 178.32.94.247
[1:00] <gregaf> Connecting to daevel.fr|178.32.94.247|:443... connected.
[1:00] <gregaf> ERROR: certificate common name `www.daevel.fr' doesn't match requested host name `daevel.fr'.
[1:00] <gregaf> To connect to daevel.fr insecurely, use `--no-check-certificate'.
[1:00] <Kioob> great :/
[1:00] <Kioob> thantks !
[1:03] <gregaf> yep
[1:03] <Kioob> (my wget accept the certificate, like iceweasel too)
[1:03] <gregaf> Kioob: 23:45 of which day?
[1:03] <Kioob> 2013-01-11
[1:04] <gregaf> (I don't know much about certs, but I assume it's about the wildcard matching due to presence/absence of "www")
[1:04] <Kioob> (it's not a wildcard)
[1:07] <gregaf> dunno then; really not my area
[1:08] <gregaf> Kioob: remind me what version this is?
[1:08] <Kioob> 0.55
[1:09] <gregaf> okay
[1:09] <gregaf> I've got to tell you, this really looks normal
[1:09] <gregaf> do you have corresponding OSD logs of any depth?
[1:09] <gregaf> (the one from yesterday with the messages doesn't overlap with this monitor log)
[1:10] <Kioob> I have log of one of the OSD, with �debug ms = 1�
[1:10] <gregaf> which OSD?
[1:10] <Kioob> osd.12
[1:12] <Kioob> oh, the �problem� was near 23h15 sorry. I shutdown OSD at 23h40
[1:14] * allsystemsarego (~allsystem@5-12-241-245.residential.rdsnet.ro) Quit (Quit: Leaving)
[1:18] <Kioob> then https://daevel.fr/osd.12.log.gz, for today's logs of that OSD
[1:19] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[1:19] <gregaf> Kioob: hmmm, that OSD doesn't appear to have led any scrubbing between 23:06 and 23:28, but maybe it followed somebody
[1:21] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[1:22] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[1:23] <Kioob> ok :/
[1:24] * noob2 (~noob2@pool-71-244-111-36.phlapa.fios.verizon.net) has joined #ceph
[1:26] * noob2 (~noob2@pool-71-244-111-36.phlapa.fios.verizon.net) Quit ()
[1:26] <Kioob> both time I add the problem, there was the PG 3.64 scrubbing, so I will add logs on one of its OSD [10,23,31], for the next time...
[1:28] <Kioob> thanks for your time gregaf
[1:29] * jlogan (~Thunderbi@2600:c00:3010:1:9cc3:821f:978c:5b0b) Quit (Ping timeout: 480 seconds)
[1:30] <Kioob> Good night ceph ;)
[1:33] <nwat> gregaf: mind reviewing some simple stuff in wip-osx-upstream sometime?
[1:33] <gregaf> nwat: I'd love to when I get the time, but… ;)
[1:34] <nwat> gregaf: sure, i just meant whenever :)
[1:36] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[1:36] * nwat (~Adium@soenat3.cse.ucsc.edu) Quit (Quit: Leaving.)
[1:50] <gregaf> Kioob, okay, there is more here than I'd expected; this log shows that when it does a scrub it's sending over about 8 MB every .5-3 seconds for a good long while
[1:50] <gregaf> what are you using your cluster for again? rbd, right?
[1:50] <gregaf> are you taking snapshots?
[1:54] * xmltok (~xmltok@pool101.bizrate.com) Quit (Ping timeout: 480 seconds)
[1:55] <paravoid> what would you suggest to do to speed up small writes performance?
[1:55] <paravoid> like 10kb objects
[1:56] * dpippenger (~riven@cpe-76-166-221-185.socal.res.rr.com) has joined #ceph
[1:58] <dec> nhm: you still around?
[1:58] <dec> looking for advice on how to debug these 0.56.1 performance problems
[1:59] <paravoid> dec: what kind of problems?
[2:00] <dec> seems that the number of write ops and write latency sky rocketed on our OSD disks after the upgrade: http://i.imgur.com/hIDa2.png http://i.imgur.com/K0BE4.png
[2:00] <dec> seems to be causing lots of iowait / poor latency on the VMs running from RBD
[2:02] <gregaf> I suspect nhm is off for the evening, although I think he occasionally pokes his head in on weekends
[2:04] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[2:05] <dec> ok, thanks
[2:07] * jjgalvez1 (~jjgalvez@12.248.40.138) Quit (Quit: Leaving.)
[2:08] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[2:10] <paravoid> huh, interesting
[2:11] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[2:12] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[2:12] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) has joined #ceph
[2:16] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[2:16] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit (Remote host closed the connection)
[2:17] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[2:18] * gaveen (~gaveen@112.135.143.71) Quit (Ping timeout: 480 seconds)
[2:19] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:22] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[2:22] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[2:24] * sleinen1 (~Adium@2001:620:0:26:a9f9:9e00:60e9:2327) Quit (Quit: Leaving.)
[2:25] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit ()
[2:25] * miroslav (~miroslav@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[2:27] * The_Bishop (~bishop@2001:470:50b6:0:2d34:6844:9a8f:4304) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[2:31] <dec> I've seen people debugging slow requests by looking at in-flight operations - how is this done?
[2:33] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[2:44] * ircolle (~ircolle@c-67-172-132-164.hsd1.co.comcast.net) Quit (Quit: Leaving.)
[2:48] * danieagle (~Daniel@177.97.248.250) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[2:48] * via (~via@smtp2.matthewvia.info) Quit (Ping timeout: 480 seconds)
[2:49] <dmick> dec: http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/7
[2:49] <dmick> argh
[2:49] <dmick> http://www.sebastien-han.fr/blog/2012/08/14/ceph-admin-socket/
[2:50] <dec> dmick: oh, cool. thanks.
[2:52] * loicd (~loic@2a01:e35:2eba:db10:120b:a9ff:feb7:cce0) Quit (Quit: Leaving.)
[3:01] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[3:04] <paravoid> wow that's a lot of metrics
[3:08] * The_Bishop (~bishop@f052096120.adsl.alicedsl.de) has joined #ceph
[3:11] <dmick> paravoid: the admin socket?
[3:11] <dmick> yeah, there are a few stats :)
[3:11] <paravoid> yeah
[3:11] <paravoid> I'm seeing terrible performance with small files but I have no baseline to compare it to
[3:11] <paravoid> like a previous version
[3:12] <paravoid> and I also have evidence of the controllers being crappy
[3:14] * gaveen (~gaveen@112.135.151.243) has joined #ceph
[3:18] * via (~via@smtp2.matthewvia.info) has joined #ceph
[3:21] <Kioob> (01:50:38) gregaf: what are you using your cluster for again? rbd, right? <=== yes RBD, the kernel version. And no, I haven't any snapshot for now
[3:34] * pedahzur (~jkugler@216-67-98-32.static.acsalaska.net) has joined #ceph
[3:37] <pedahzur> Reading up on ceph. Thinking about a deployment. I'm looking for clustering file system functionality (i.e. storage accessible by several hosts at once). If I understand correctly, I'll need to use CephFS, correct? If I formatted a ceph block store as, say, ext4, then mounted it several hosts, I'd have serious issues, yes?
[3:38] <dmick> yes
[3:38] <dmick> it's conceivable you could export the storage with another shared FS, but CephFS is one
[3:38] <pedahzur> I'd rather have to maintain NFS on top of Ceph, if I could avoid it. :)
[3:39] <pedahzur> So, reading the web site, it still says CephFS is somewhat unstable. OK...ustable as in, it's pretty good, but it may see a lot of changes, and we're still working out the corner cases, or unstable as in, "OH NOES! Your data is going to go down in flames?" :)
[3:39] <dmick> if you go through kernel rbd, you can use/export however you like of course.
[3:40] <dmick> pedahzur: single-active-MDS configs are workable (standby MDSes, not simultaneously active)
[3:43] * LeaChim (~LeaChim@b0fadd12.bb.sky.com) Quit (Ping timeout: 480 seconds)
[3:46] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[3:46] <pedahzur> Still getting the terminology down: MDS is meta-data server, right? You mean for CephFS, it can use one MDS at a time, and will fail-over to a backup MDS if need be? Ah...only CephFS requires an MDS, right? So, do the standby MDS keep up-to-date in case they have to go active?
[3:48] <pedahzur> (Yes, I need to read more) :)
[3:49] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[3:50] <dmick> yes metadataserver, yes only for CephFS, and yes autofailover I believe
[3:52] * BManojlovic (~steki@85.222.220.181) Quit (Quit: Ja odoh a vi sta 'ocete...)
[3:53] <dmick> and the data is stored in RADOS itself, so the state is effectively shared (i.e. the standby MDSes don't do anything special to stay uptodate)
[3:56] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[3:57] <pedahzur> Ah, understood.
[3:57] <pedahzur> Thanks for the answers...that helps. So, here is my use case. Maybe you can say "Cool!" or "You're off your rocker and asking for trouble". I have four machines. Each machine has eight 500GB drives. Each machine will run 2VM with 100GB images (raw partitions), per drive. So, each machine will have 300GB * 8 free. That gives me 9.6GB (raw, nominal) of free space to play with. I want to set up a shared file system for the VMs since doing
[3:57] <pedahzur> so will reduce some of the VM's workload (the VMs are build servers, so sharing some of the built files reduces overall work). I want to use that free space to be used by Ceph, and have the VM's mount the shared space via CephFS. This is not a high (or constant) I/O setup. I'm looking for easy clustered space, and automated replication/fail over for the shared space. Good idea? Bad idea? Fair-to-middling? :)
[4:03] * Cube1 (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[4:04] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[4:17] * Ryan_Lane (~Adium@216.38.130.165) Quit (Quit: Leaving.)
[4:26] * mattbenjamin (~matt@75.45.227.140) has joined #ceph
[4:26] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[4:39] <dmick> pedahzur: that doesn't sound overly crazy to me
[4:42] <dmick> That 9.6TB will be halved-and-overheaded-more, of course, for the reliability, and you'll want to think about journal placement vs. reliability (since a failed journal results in a failed OSD, but you probably don't want journals on the same drive as the OSD they're serving for performance, you might need to think carefully about failure domains)
[4:42] <dmick> but it's certainly doable and will be lots more reliable than raw disks
[4:55] <pedahzur> Yeah, I know I won't get 9.6TB...wasn't expecting to. :) Just saying how much space I have to play with. Thanks. Glad to know It's not a terribly crazy idea.
[5:13] <mikedawson> If i want to test the fix at http://www.tracker.newdream.net/issues/3770 sometime soon, is there an automatic build available or am I on my own to compile?
[5:31] <dmick> that's in master, so there's an automatic build that's the bleeding edge
[5:32] <dmick> presumably soon it will appear in a 0.56.x branch, but not 100% clear
[5:33] <dmick> if you want to try that, you can find master packages using http://ceph.com/docs/master/install/debian/#development-testing-packages
[5:40] <mikedawson> thanks dmick
[5:40] <dmick> np
[5:41] <mikedawson> do you know if the fix prevents the issue in the future or will repair an OSD currently suffering the issue?
[5:44] <dmick> not for certain, but it seems to me with a half-understanding of the issue that it should change the OSD such that this condition is not fatal on startup
[5:44] <dmick> so while it may still happen, the OSD startup/catchup will handle it correctly
[5:44] <dmick> that is, the condition itself is not abnormal
[5:45] <mikedawson> dmick: sounds good, I'll give the dev repo a test and see if I can resurrect my testing cluster
[5:45] <dmick> gl
[5:57] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[5:57] <mikedawson> dmick: should echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/wip_3770 $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list get me something on Quantal?
[5:57] <dmick> you don't want wip_3770; you want master
[5:58] <mikedawson> switched to master ... 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[5:58] <mikedawson> hrm
[5:58] <dmick> 0.56-219-g7ea5d84-1quantal is there, or should be
[5:59] <dmick> (you did run apt-get update of course?)
[5:59] <mikedawson> echo deb http://gitbuilder.ceph.com/ceph-deb-$(lsb_release -sc)-x86_64-basic/ref/master $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
[5:59] <mikedawson> apt-get update && sudo apt-get -V dist-upgrade
[6:00] <dmick> should work. Maybe check showpkg etc. to see what it's claiming is available
[6:05] <mikedawson> dmick: ceph 0.56.1-1quantal newer than version in archive
[6:05] <dmick> oh does the .1 throw it?
[6:05] <dmick> or is it really older?
[6:06] <dmick> no, those were built today. I can never remember the twisty details of package versions but it could be
[6:08] <dmick> yes, 0.56.1 is later that 0.56. feh.
[6:08] <dmick> that's why you really want a next build I guess :)
[6:12] <dmick> I'd force the update to that specific version; I don't see a much better way
[6:13] <dmick> i.e. apt-get install=0.56-219-g7ea5d84-1quantal
[6:13] <dmick> (install ceph=, etc. of course I mean)
[6:16] <mikedawson> still crashes
[6:17] <mikedawson> ceph -v is ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
[6:17] <dmick> that's not the right one
[6:17] <mikedawson> log shows ceph version 0.56-219-g7ea5d84 (7ea5d84fa3d0ed3db61eea7eb9fa8dbee53244b6)
[6:17] <mikedawson> that doesn't seem right
[6:17] <dmick> that's the right one
[6:17] <dmick> hm
[6:17] <dmick> did you update all the packages?
[6:17] <dmick> and then bounce the daemons?
[6:18] <dmick> (the ceph utility is in ceph-common, for instance)
[6:18] <mikedawson> I did apt-get remove ceph && apt-get install ceph && service ceph restart
[6:19] <mikedawson> guess that didn't get all packages
[6:20] <phantomcircuit> hmm interesting the activity from ceph-mon is non trivial i got a nice performance boost putting it on ssds
[6:22] <dmick> mike: right; ceph isn't a metapackage
[6:22] <mikedawson> dmick: trying apt-get install ceph-common=0.56-219-g7ea5d84-1quantal ceph-fs-common=0.56-219-g7ea5d84-1quantal ceph-fuse=0.56-219-g7ea5d84-1quantal librbd1=0.56-219-g7ea5d84-1quantal librados2=0.56-219-g7ea5d84-1quantal
[6:23] <dmick> that's many of them
[6:24] <mikedawson> still crashes
[6:24] <mikedawson> 2013-01-12 00:23:53.732911 7fdda9821780 -1 osd/OSD.cc: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fdda9821780 time 2013-01-12 00:23:53.731471 osd/OSD.cc: 4432: FAILED assert(_get_map_bl(epoch, bl))
[6:24] <mikedawson> dmick: any more I should pick up?
[6:26] <dmick> just to be certain, try this:
[6:26] <dmick> for i in $(ceph osd ls); do ceph tell osd.$i version; done
[6:27] <mikedawson> comical
[6:27] <mikedawson> ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
[6:27] <mikedawson> ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
[6:27] <mikedawson> osd.2 is not up
[6:27] <mikedawson> ceph version 0.56-219-g7ea5d84 (7ea5d84fa3d0ed3db61eea7eb9fa8dbee53244b6)
[6:27] <mikedawson> osd.4 is not up
[6:27] <mikedawson> ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
[6:27] <mikedawson> osd.6 is not up
[6:27] <mikedawson> ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
[6:27] <dmick> erm...
[6:27] <mikedawson> 2 and 3 are on the box I'm working on right now. 2 suffers the issue (and therefore won't start)
[6:27] <dmick> oh and you only tried to upgrade those two, I see
[6:28] <mikedawson> 4 and 6 also suffer the issue
[6:28] <mikedawson> didn't try to upgrade node1 (all working), node3, or node4 yet
[6:28] <dmick> well the fact that osd.3 is up with the right version means osd.2 would've been running that one too
[6:28] <dmick> so now I'm afraid it's back to samf
[6:29] <mikedawson> samf?
[6:30] <dmick> sorry wrong handle
[6:30] <dmick> sjust
[6:30] <mikedawson> yep
[6:30] <mikedawson> is a comment on the tracker the best feedback?
[6:30] <dmick> that'll do, yeah
[6:31] <mikedawson> will do
[6:31] <dmick> can also drop him a note at samuel.just@inktank.com if you want redundancy
[6:31] * zK4k7g (~zK4k7g@digilicious.com) Quit (Quit: Leaving.)
[6:39] <mikedawson> dmick: went with the email only at this point in case the patch isn't intended to fix an affected osd. Thanks for your help!
[6:39] <dmick> ok. sorry it didn't work out, and gl.
[6:39] <mikedawson> no worries
[7:00] <dmick> mikedawson: is your failure from an assert?
[7:01] <mikedawson> y
[7:01] <mikedawson> 2013-01-12 00:23:53.732911 7fdda9821780 -1 osd/OSD.cc: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fdda9821780 time 2013-01-12 00:23:53.731471 osd/OSD.cc: 4432: FAILED assert(_get_map_bl(epoch, bl))
[7:01] <dmick> I should say
[7:01] <dmick> is that the same assert as before the 'fix'?
[7:01] <dmick> bug says so. never mind.
[7:02] <mikedawson> seems identical
[7:02] <dmick> found a similar-sounding assert in our test cluster; I'll bring it to Sam's attention as well, may be connected
[7:12] * dmick (~dmick@2607:f298:a:607:9528:e89b:6c31:616) Quit (Quit: Leaving.)
[7:24] * gaveen (~gaveen@112.135.151.243) Quit (Ping timeout: 480 seconds)
[7:38] * Pagefaulted (~AndChat73@c-67-168-132-228.hsd1.wa.comcast.net) has joined #ceph
[7:39] * aviziva (~aviziva@c-24-6-144-43.hsd1.ca.comcast.net) has joined #ceph
[7:39] * aviziva (~aviziva@c-24-6-144-43.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[7:40] <Pagefaulted> Are there security benefits to having the osd replication network non routed and separated from client access network?
[7:42] <Pagefaulted> Or does it just add unneeded complexity?
[7:46] * mattbenjamin (~matt@75.45.227.140) Quit (Quit: Leaving.)
[7:53] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[8:02] <phantomcircuit> Pagefaulted, i believe the recommended hardware is dual gig nics
[8:02] <phantomcircuit> with clients accessing the osd on one and the osd replication on the other
[8:03] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[8:05] <Pagefaulted> What about bonding two NICs with sub interfaces on each vlan?
[8:13] <Pagefaulted> the idea would be to maximise the limited 2, 1gig links.
[8:14] <Pagefaulted> I'm using some old hardware with only 1 gig NICs.
[8:54] * loicd (~loic@magenta.dachary.org) has joined #ceph
[9:24] * dec_ (~dec@ec2-54-251-62-253.ap-southeast-1.compute.amazonaws.com) has joined #ceph
[9:25] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[9:25] * loicd (~loic@2a01:e35:2eba:db10:98ab:4fd4:388e:6140) has joined #ceph
[9:26] * dec (~dec@ec2-54-251-62-253.ap-southeast-1.compute.amazonaws.com) Quit (Ping timeout: 480 seconds)
[9:44] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Read error: Connection reset by peer)
[9:48] * The_Bishop_ (~bishop@e179005027.adsl.alicedsl.de) has joined #ceph
[9:53] * The_Bishop (~bishop@f052096120.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[9:55] * nyeates (~nyeates@pool-173-59-239-231.bltmmd.fios.verizon.net) Quit (Quit: nyeates)
[9:58] * tnt (~tnt@86.188-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[10:06] * LeaChim (~LeaChim@b0fadd12.bb.sky.com) has joined #ceph
[10:22] * madkiss (~madkiss@p57A1DB13.dip.t-dialin.net) has joined #ceph
[10:25] * stxShadow (~Jens@ip-178-201-147-146.unitymediagroup.de) has joined #ceph
[10:35] <madkiss> moin moin
[10:35] <stxShadow> moin
[11:00] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[11:03] * sleinen1 (~Adium@2001:620:0:25:a90a:dc6:1a92:b197) has joined #ceph
[11:08] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[11:16] * pedahzur (~jkugler@216-67-98-32.static.acsalaska.net) Quit (Quit: Konversation terminated!)
[11:52] * MK_FG (~MK_FG@00018720.user.oftc.net) Quit (Ping timeout: 480 seconds)
[12:01] * madkiss (~madkiss@p57A1DB13.dip.t-dialin.net) Quit (Quit: Leaving.)
[12:01] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[12:01] * loicd (~loic@2a01:e35:2eba:db10:98ab:4fd4:388e:6140) Quit (Quit: Leaving.)
[12:02] * loicd (~loic@magenta.dachary.org) has joined #ceph
[12:07] * MK_FG (~MK_FG@00018720.user.oftc.net) has joined #ceph
[12:12] * mattbenjamin (~matt@75.45.227.140) has joined #ceph
[12:16] * KindOne (~KindOne@50.96.83.114) Quit (Remote host closed the connection)
[12:26] * KindOne (~KindOne@50.96.83.114) has joined #ceph
[12:39] * MKFG (~MK_FG@188.226.51.71) has joined #ceph
[12:44] * MK_FG (~MK_FG@00018720.user.oftc.net) Quit (Ping timeout: 480 seconds)
[12:44] * MKFG is now known as MK_FG
[12:52] * ScOut3R (~ScOut3R@2E6BA080.dsl.pool.telekom.hu) has joined #ceph
[12:55] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Quit: Leaving.)
[13:07] * madkiss (~madkiss@p57A1DB13.dip.t-dialin.net) has joined #ceph
[13:20] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[13:25] * benner_ (~benner@193.200.124.63) has joined #ceph
[13:25] * benner (~benner@193.200.124.63) Quit (Read error: Connection reset by peer)
[14:31] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[14:39] * ScOut3R (~ScOut3R@2E6BA080.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[14:45] * ScOut3R (~ScOut3R@2E6BA080.dsl.pool.telekom.hu) has joined #ceph
[14:46] * madkiss (~madkiss@p57A1DB13.dip.t-dialin.net) Quit (Quit: Leaving.)
[14:47] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[14:56] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[15:04] * yanzheng (~zhyan@134.134.139.76) has joined #ceph
[15:06] * Tribaal (uid3081@id-3081.hampstead.irccloud.com) Quit (Quit: Planned maintenance, back soon)
[15:12] * kylehutson (~kylehutso@dhcp231-11.user.cis.ksu.edu) Quit (Read error: Operation timed out)
[15:28] <phantomcircuit> hmm
[15:29] <nhm> hmm indeed!
[15:33] <phantomcircuit> partition osd filestore is on is getting in excess of 100 write operations/second
[15:34] <phantomcircuit> i dont think it's very happy
[15:39] <phantomcircuit> interesting the only code path that sets force_sync when you have a journal is only called on unmount
[15:41] <phantomcircuit> which begs the question why so many ops
[15:42] <phantomcircuit> ooh
[15:42] <phantomcircuit> it's the io size
[15:42] <phantomcircuit> tons of tiny ios
[15:47] * tnt_ (~tnt@216.186-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[15:50] * tnt (~tnt@86.188-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[15:57] * Tribaal (uid3081@id-3081.tooting.irccloud.com) has joined #ceph
[15:59] * gaveen (~gaveen@112.135.154.176) has joined #ceph
[16:09] * yanzheng (~zhyan@134.134.139.76) Quit (Remote host closed the connection)
[16:19] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[16:24] * sleinen (~Adium@2001:620:0:25:a90a:dc6:1a92:b197) has joined #ceph
[16:24] * sleinen1 (~Adium@2001:620:0:25:a90a:dc6:1a92:b197) Quit (Read error: Connection reset by peer)
[16:26] * yanzheng (~zhyan@134.134.139.74) has joined #ceph
[16:30] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) has joined #ceph
[16:32] * yanzheng (~zhyan@134.134.139.74) Quit (Remote host closed the connection)
[16:36] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[16:36] * loicd (~loic@magenta.dachary.org) has joined #ceph
[16:54] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[17:03] * tsygrl (~tsygrl314@c-75-68-140-25.hsd1.vt.comcast.net) has joined #ceph
[17:09] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[17:10] * loicd (~loic@magenta.dachary.org) has joined #ceph
[17:11] * tjikkun_ (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[17:12] * tjikkun_ (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit ()
[17:12] * ScOut3R (~ScOut3R@2E6BA080.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[17:14] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[17:14] * stxShadow (~Jens@ip-178-201-147-146.unitymediagroup.de) Quit (Read error: Connection reset by peer)
[17:20] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[17:20] * loicd (~loic@magenta.dachary.org) has joined #ceph
[17:22] * jluis (~JL@89.181.154.239) has joined #ceph
[17:24] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[17:28] * joao (~JL@89-181-159-29.net.novis.pt) Quit (Ping timeout: 480 seconds)
[17:53] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:53] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[17:54] * nwat (~Adium@c-50-131-197-174.hsd1.ca.comcast.net) has left #ceph
[18:13] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) Quit (Quit: Leaving.)
[18:49] * tziOm (~bjornar@ti0099a340-dhcp0628.bb.online.no) Quit (Quit: Leaving)
[19:06] * sleinen1 (~Adium@2001:620:0:25:8c15:f84f:db60:4df5) has joined #ceph
[19:10] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[19:11] <phantomcircuit> how full does the journal need to be before it starts to block?
[19:11] * sleinen (~Adium@2001:620:0:25:a90a:dc6:1a92:b197) Quit (Ping timeout: 480 seconds)
[19:11] * DLange (~DLange@dlange.user.oftc.net) Quit (Quit: $funnymessagegoeshere)
[19:14] * nwat (~Adium@soenat3.cse.ucsc.edu) has joined #ceph
[19:14] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[19:36] * Pagefaulted (~AndChat73@c-67-168-132-228.hsd1.wa.comcast.net) Quit (Quit: Bye)
[19:36] * Pagefaulted (~AndChat73@c-67-168-132-228.hsd1.wa.comcast.net) has joined #ceph
[19:41] * jjgalvez (~jjgalvez@ec2-54-235-219-17.compute-1.amazonaws.com) has joined #ceph
[19:45] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[19:45] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:46] * loicd (~loic@magenta.dachary.org) Quit ()
[20:00] * gaveen (~gaveen@112.135.154.176) Quit (Ping timeout: 480 seconds)
[20:11] * CloudGuy (~CloudGuy@62.140.137.136) has joined #ceph
[20:13] * The_Bishop_ (~bishop@e179005027.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[20:26] * The_Bishop (~bishop@e179016199.adsl.alicedsl.de) has joined #ceph
[20:30] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[20:34] * The_Bishop (~bishop@e179016199.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[20:41] * The_Bishop (~bishop@e179019173.adsl.alicedsl.de) has joined #ceph
[20:43] * mattbenjamin (~matt@75.45.227.140) Quit (Read error: Connection reset by peer)
[20:45] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[20:46] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[20:52] * jjgalvez (~jjgalvez@ec2-54-235-219-17.compute-1.amazonaws.com) Quit (Quit: Leaving.)
[20:55] * The_Bishop (~bishop@e179019173.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[20:56] * CloudGuy (~CloudGuy@62.140.137.136) Quit (Ping timeout: 480 seconds)
[20:58] * The_Bishop (~bishop@e179019173.adsl.alicedsl.de) has joined #ceph
[21:02] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[21:02] * markl (~mark@tpsit.com) Quit (Ping timeout: 480 seconds)
[21:23] * The_Bishop_ (~bishop@e179016068.adsl.alicedsl.de) has joined #ceph
[21:24] * The_Bishop__ (~bishop@e179010188.adsl.alicedsl.de) has joined #ceph
[21:26] * The_Bishop_ (~bishop@e179016068.adsl.alicedsl.de) Quit (Read error: Connection reset by peer)
[21:27] * The_Bishop_ (~bishop@e179000050.adsl.alicedsl.de) has joined #ceph
[21:31] * The_Bishop (~bishop@e179019173.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[21:33] * The_Bishop__ (~bishop@e179010188.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[21:37] * danieagle (~Daniel@177.97.250.144) has joined #ceph
[21:46] * The_Bishop__ (~bishop@e179004066.adsl.alicedsl.de) has joined #ceph
[21:49] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:50] * allsystemsarego (~allsystem@5-12-241-245.residential.rdsnet.ro) has joined #ceph
[21:54] * The_Bishop_ (~bishop@e179000050.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[21:54] * Cube (~Cube@184-215-241-155.pools.spcsdns.net) has joined #ceph
[21:58] * The_Bishop__ (~bishop@e179004066.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[22:04] * gluffis (~gluffis@castro.mean.net) Quit (Read error: Connection reset by peer)
[22:11] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[22:11] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[22:12] * danieagle (~Daniel@177.97.250.144) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[22:16] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:17] * loicd (~loic@magenta.dachary.org) Quit ()
[22:25] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:47] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[22:50] * allsystemsarego_ (~allsystem@188.27.166.150) has joined #ceph
[22:58] * allsystemsarego (~allsystem@5-12-241-245.residential.rdsnet.ro) Quit (Ping timeout: 480 seconds)
[23:06] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[23:08] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:18] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[23:23] * sleinen1 (~Adium@2001:620:0:25:8c15:f84f:db60:4df5) Quit (Quit: Leaving.)
[23:30] * nwat (~Adium@soenat3.cse.ucsc.edu) Quit (Quit: Leaving.)
[23:33] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) has joined #ceph
[23:46] * CloudGuy_ (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) has joined #ceph
[23:46] * CloudGuy (~CloudGuy@5356416B.cm-6-7b.dynamic.ziggo.nl) Quit (Read error: Connection reset by peer)
[23:51] * The_Bishop (~bishop@e177090243.adsl.alicedsl.de) has joined #ceph
[23:54] * wschulze (~wschulze@cpe-98-14-23-162.nyc.res.rr.com) Quit (Quit: Leaving.)
[23:59] * Cube (~Cube@184-215-241-155.pools.spcsdns.net) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.