#ceph IRC Log

Index

IRC Log for 2012-10-05

Timestamps are in GMT/BST.

[0:02] * cblack101 (86868b4c@ircip2.mibbit.com) has joined #ceph
[0:03] * gregorg_taf (~Greg@78.155.152.6) Quit (Ping timeout: 480 seconds)
[0:03] <Kioob> I understand nhm
[0:05] * psomas (~psomas@inferno.cc.ece.ntua.gr) has joined #ceph
[0:05] <Kioob> so, "sequential" writes of 8Ko can be a problem ?
[0:05] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[0:06] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[0:07] <Kioob> maybe I should try first DRBD+Btrfs to see if the CoW way can solved that first problem
[0:08] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit ()
[0:08] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[0:08] <nhm> Kioob: rbd cache will definitely help. Those transaction log writes though are O_SYNC and by default it looks like it does 512b writes.
[0:10] <Kioob> but rdb cache is enabled in the 0.48 version ?
[0:11] <Kioob> mmm yes, since 0.46
[0:11] <Kioob> that write-back feature is safe ?
[0:11] <nhm> I think so, ask joshd. ;)
[0:12] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[0:12] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[0:13] <gregaf> yes, it's safe and respects all flush, sync, etc commands
[0:13] <Kioob> ok, I'm reading the doc. It follow barrier & flush requets, so it's wonderfull :)
[0:14] <Kioob> (just a note, I also have a small 512MB of write-back on the sata controller, with a BBU)
[0:14] <nhm> Kioob: which controller?
[0:15] <Kioob> LSI MegaRAID SAS 9260-4i
[0:15] <nhm> Kioob: How many drives per controller?
[0:16] <Kioob> 8 to 12...
[0:16] <Kioob> (it's a bottleneck)
[0:16] <tren> sas2 backplane?
[0:16] <nhm> Ok, so expanders between the drives and the controller.. Are these SATA disks?
[0:16] <Kioob> nhm: SAS disks, and SATA SSD
[0:17] <Kioob> tren ?
[0:17] <nhm> Kioob: Ok, the SATA disks may force you into 3.0Gb
[0:17] <darkfader> Kioob: thanks for writing along while you read the async mode. i had never planned trying it since i thought it were more linuxy and not respecting the syncs etc. haha
[0:17] <tren> Kioob: Is the sas expander sas1 or sas2?
[0:18] <nhm> Kioob: that combo could also be slowing you down. expanders (especially with SATA disks) can be rather problematic.
[0:18] <Kioob> tren : I really don't know
[0:19] <tren> nhm: we run sata disks on sas expanders with little difficulty. The key was ensuring current kernel
[0:19] <Kioob> nhm: yes... I already disable the hardware RAID6 because it was a lot slower than the software one.... I suppose the card is the problem, but don't really understand why
[0:20] <tren> Kioob: Have you tried switching to a 9211-4i instead
[0:21] <Kioob> no I didn't tried.
[0:21] <tren> Kioob: Instead of using a raid card and exporting individual disks, try using a SAS HBA like the 9211-4i
[0:21] <tren> Kioob: Plus they're FAR cheaper ;)
[0:21] <nhm> tren: We've got some 12 drive systems with SAS2108 controllers and expanders. We've had a lot of trouble getting good performance out of them. Another system with various other controllers and no expanders does much better.
[0:22] <Kioob> In fact I'm limited to a (really) short list of hardware, and didn't find better card
[0:22] <nhm> tren: Some of the highest performing controllers for Ceph are cheap SAS controllers.
[0:22] <Kioob> (and the current list only have that card :()
[0:22] <nhm> At least when paired with btrfs on a fresh filesystem.
[0:23] <tren> nhm: We use 9211's for most of our SAS use. Unfortunately these new dell servers we got have 2008's in IR with the raid key installed. So there's NO option to flash these to IT mode
[0:23] <nhm> tren: you can't flash the H200s to IT mode if the raid key is installed? That's really good to know.
[0:23] <nhm> tren: I was planning on trying to get some H200/H310s for our Dell nodes.
[0:24] <nhm> tren: Not sure about the Dell firmware on the H710, but on supermicro's SAS2208 firmware you can do JBOD with IR.
[0:26] * tren (~Adium@184.69.73.122) Quit (Read error: Connection reset by peer)
[0:26] * tren (~Adium@184.69.73.122) has joined #ceph
[0:26] <tren> stupid internet. :/
[0:27] <tren> our office internet is having issues
[0:27] <nhm> tren: I was planning on trying to get some H200/H310s for our Dell nodes.
[0:27] <nhm> tren: Not sure about the Dell firmware on the H710, but on supermicro's SAS2208 firmware you can do JBOD with IR.
[0:27] <gregaf> tren: which client are you using? ceph-fuse or kernel?
[0:28] <nhm> tren: it's good to know that the H200 can't take IT firmware if the raid key is installed.
[0:28] <nhm> tren: that's exactly what I wanted to do.
[0:30] <Kioob> last question : which kernel version do you recommends to use rbd, on OSD ? and on client ?
[0:35] * tren (~Adium@184.69.73.122) Quit (Ping timeout: 480 seconds)
[0:35] <gregaf> if you're not using btrfs it doesn't matter on the OSD (assuming it's not ancient); and if you're using librbd it doesn't matter much at all for the clients
[0:36] <Kioob> I will use btrfs, so I should use a recent kernel I suppose
[0:37] * tren (~Adium@184.69.73.122) has joined #ceph
[0:37] * rweeks staples tren's internet connection down
[0:38] <Kioob> to create a rbd, the librbd is used ?
[0:38] <gregaf> tren: which client are you using? ceph-fuse or kernel?
[0:38] <tren> gregaf: ceph-fuse
[0:38] * gregorg (~Greg@78.155.152.6) has joined #ceph
[0:38] <tren> gregaf: kernel client no good on osd nodes :)
[0:38] <gregaf> also v0.52, I assume?
[0:39] <tren> gregaf: yes, and kernel 3.5.3
[0:39] <gregaf> k, thanks
[0:39] <tren> gregaf: soon to be kernel 3.6
[0:40] <tren> nhm: You can reflash a H200 to be IT mode. I've done it with a couple of them
[0:40] <tren> nhm: Unfortunately, their newer raid cards you cannot
[0:41] <tren> nhm: The new PERC H310 as far as I know, does not have a path to flash to IT mode.
[0:41] * BManojlovic (~steki@212.200.240.160) Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:42] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[0:43] <Tv_> soo.. ceph-osd -i 9 --get-journal-fsid
[0:43] <Tv_> b755fb56-31b8-4e48-a8a6-6f57758ca64c
[0:43] <Tv_> 2012-10-04 22:43:10.430733 7f963367e780 -1 journal FileJournal::_open: aio not supported without directio; disabling aio
[0:43] <Tv_> why is it complaining about aio and how can i make it happier (or just shut up)
[0:44] <dmick> default constructor sets aio = true
[0:45] <dmick> and directio = false
[0:45] <dmick> hm
[0:46] <dmick> (and the warning is just based on the state of those instance vars)
[0:46] <nhm> tren: Is that using the SAS2308 chip? I've actually got one of those with IT firmware.
[0:46] <gregaf> dmick: they should also be set up based on the config values, which are different from that, right?
[0:46] <nhm> tren: I thought Dell claimed the SAS2308 did JBOD mode though.
[0:46] <Tv_> dmick: the osd in daemon has no such messages
[0:46] <nhm> er the H310 rather.
[0:47] <nhm> Maybe they rolled it into the IR firmware?
[0:47] <dmick> gregaf: possible, just starting from the message and working outward
[0:47] <gregaf> is Tv_doing something that's bypassing normal config parsing?
[0:48] <dmick> and yes, the default config options are the reverse, amusingly
[0:48] <Tv_> gregaf: running ceph-osd --get-journal-fsd
[0:48] <Tv_> gregaf: so non-daemon mode
[0:49] <Tv_> --journal-aio=false --journal-dio=false has no effect on message
[0:50] <dmick> --get-journal-fs*i*d, and yeah, same for me
[0:50] <gregaf> oh, looks like get-journal-fsid is creating a FileJournal inline with a shorter constructor
[0:50] <gregaf> bet the regular path creates one that references the config options properly
[0:51] <Tv_> ah
[0:51] <gregaf> so, tell sjust or mikeryan to make it less stupid, or do so yourself
[0:52] <sjust> ugh
[0:52] <sjust> bug it
[0:52] <dmick> - FileJournal j(fsid, 0, 0, path.c_str());
[0:52] <dmick> + FileJournal j(fsid, 0, 0, path.c_str(), false, false);
[0:53] <dmick> ^ worth everything you paid for it
[0:53] <tren> nhm: RAID JBOD != HBA
[0:53] <dmick> tren: on a reasonable RAID controller it ought to be, but LSI are not reasonable
[0:53] <tren> nhm: They're 2008 based. That's the crazy thing
[0:54] <tren> dmick: I haven't really found any raid controller that'll export individual disks well. The worst is 3ware IMHO.
[0:57] <dmick> tren: I know; "reasonable" isn't something I think necessarily exists :)
[0:58] <tren> dmick: You pay more for reasonable ;)
[0:58] <dmick> would that that's all it took
[0:59] <nhm> tren: Is that true on the 2208's JBOD mode?
[1:00] <tren> nhm: I'm not sure, we don't use 2208's in that fashion. We only use 2008's in IR mode when we have to (aka, dell did it to us)
[1:00] <tren> nhm: If we build the servers ourselves, we just use a HBA 9211 and save ourselves the trouble
[1:00] <nhm> tren: Yeah, with the H700 we just do single disk raid0s which is definitely not JBOD.
[1:01] <tren> nhm: The high end is their H710 now
[1:01] <tren> nhm: which we have in our MDS nodes
[1:01] <tren> nhm: sorry, mon/mds nodes
[1:01] <nhm> tren: You'll definitely be interested in the article I'm writing. I'm comparing ceph performance on 7 different controllers in various raid/jbod/pass-through modes.
[1:02] <tren> nhm: nice! are you publishing to the list when you're done? ;)
[1:02] <nhm> tren: sadly our Dell nodes don't get anywhere close to what a 9211-8i can do. :(
[1:02] <Tv_> awesome! if you hit control-d in gdisk, it goes into a busy loop reading 0 bytes from stdin and re-issuing its prompt
[1:02] <Tv_> kwa-li-tee
[1:02] <nhm> tren: Yeah, it's going on our blog, and I'll link it to the list.
[1:03] <tren> nhm: Thanks! :)
[1:04] <nhm> tren: I've got throughput results at 4k, 128k, and 4M object sizes with rados bench with XFS, BTRFS, and EXT4. I've got charts with CPU utilization numbers, queue wait times on the data disks and journal disks. I'm hoping to do some follow-up articles exploring what processes are using up CPU time, how performance and CPU usage changes during the duration of the tests, profiling results, seek behavior on the disks, etc.
[1:05] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[1:05] <tren> nhm: wow, very thorough!
[1:05] <nhm> tren: there's something like 300GB of debugging data for the tests. :)
[1:06] <nhm> tren: yeah, the first article won't go quite that deep, but I'm trying to balance making it readable while still making it interesting.
[1:06] <nhm> tren: there's just so much data...
[1:07] <tren> tren: I hear ya! But at this point any information I can get my hands on with ceph, I read ;)
[1:07] <tren> I was kinda scared to come on the irc channel actually
[1:07] <tren> but you guys have been great :)
[1:08] <nhm> tren: that's good to hear! I'm continually in awe at how smart these guys are. ;)
[1:09] <tren> :D
[1:09] <tren> so am I
[1:09] <tren> and patient
[1:09] * cblack101 (86868b4c@ircip2.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[1:23] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[1:25] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Remote host closed the connection)
[1:26] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[1:26] * jlogan1 (~Thunderbi@2600:c00:3010:1:787f:c6f:10bb:bf2) Quit (Ping timeout: 480 seconds)
[1:36] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) has left #ceph
[1:40] <sagewk> slang: there?
[1:40] * Kioob (~kioob@luuna.daevel.fr) Quit (Ping timeout: 480 seconds)
[1:40] <sagewk> slang: on the wip-client-stale: is this someone calling ->mount() twice or something? or a new client instance? i'm not sure how a new client can get a message intended for a previous client instance.. that sounds like the bug
[1:44] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[1:48] * Cube1 (~Adium@12.248.40.138) has joined #ceph
[1:53] <slang> sagewk: ok I wasn't sure
[1:54] <Tv_> sagewk: --get-journal-fsid is not very helpful -- it insist on running as one of the osds, and opening the data directory, yet the time when i really need it is before i have the data directory at hand
[1:54] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[1:54] <Tv_> sagewk: and before i know what id is correct
[1:55] <sagewk> tv_: hmm.
[1:55] <slang> it looks like the mds will fall back to just using the destination of the client if the connection is gone (when sending the stale message)
[1:55] <sagewk> tv_: that seems pretty fixable.
[1:55] <Tv_> sagewk: i think it needs to just skip more of the initialization
[1:55] <slang> does it try to use the pid?
[1:55] <sagewk> slang: yeah, but the new client should get a new address and id...
[1:55] <sagewk> slang: you're able to reproduce this?
[1:55] <sagewk> tv_: yeah
[1:56] <sagewk> tv_: i should be able to fix that up pretty easily
[1:56] <slang> in answer to your question, the stale message is arriving at a new client that hasn't yet gotten the session open reply
[1:56] <slang> sagewk: yeah it happens maybe 30% of the time
[1:57] <slang> sagewk: I can try to pare it down into a more specific test
[1:57] <sagewk> just generating the client-side log should be enough, i think.
[2:03] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) has joined #ceph
[2:07] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[2:10] * Tv_ (~tv@2607:f298:a:607:5c1e:e9a0:aa30:35e7) Quit (Quit: Tv_)
[2:14] <slang> sagewk: https://raw.github.com/gist/3837267/07801cd48778aba435e8021c22d25bdf11579086/gistfile1.txt
[2:19] * tren (~Adium@184.69.73.122) Quit (Quit: Leaving.)
[2:20] * sagelap (~sage@38.122.20.226) Quit (Ping timeout: 480 seconds)
[2:21] * edv (~edjv@107.1.75.186) Quit (Quit: Leaving.)
[2:26] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[2:27] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit ()
[2:27] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[2:32] * sagelap (~sage@100.sub-70-197-145.myvzw.com) has joined #ceph
[2:40] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[2:49] * slang (~slang@ace.ops.newdream.net) has joined #ceph
[3:00] * scuttlemonkey (~scuttlemo@38.122.20.226) Quit (Quit: This computer has gone to sleep)
[3:05] * sagelap (~sage@100.sub-70-197-145.myvzw.com) Quit (Ping timeout: 480 seconds)
[3:11] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[3:11] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[3:19] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[3:31] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[3:31] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[3:38] <slang> sagelap: it looks like maybe the -1 fix in choose_target_mds may have flushed out some other bugs
[3:40] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[3:54] * maelfius (~mdrnstm@66.209.104.107) Quit (Quit: Leaving.)
[4:06] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[4:06] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[4:08] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (Quit: Leaving.)
[4:14] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[4:30] * edv (~edjv@50-46-210-159.evrt.wa.frontiernet.net) has joined #ceph
[4:50] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[4:56] * scuttlemonkey (~scuttlemo@12.239.189.130) has joined #ceph
[4:59] * slang (~slang@ace.ops.newdream.net) Quit (Ping timeout: 480 seconds)
[4:59] * Cube1 (~Adium@12.248.40.138) Quit (Ping timeout: 480 seconds)
[5:11] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[5:12] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[5:26] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[5:29] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[5:29] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[5:30] * maelfius (~mdrnstm@pool-71-160-33-115.lsanca.fios.verizon.net) has joined #ceph
[5:36] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:45] * maelfius (~mdrnstm@pool-71-160-33-115.lsanca.fios.verizon.net) Quit (Quit: Leaving.)
[5:47] * tren (~Adium@S010658b035f97097.vc.shawcable.net) has joined #ceph
[5:57] * cephalobot` (~ceph@ps94005.dreamhost.com) has joined #ceph
[5:58] * cephalobot (~ceph@ps94005.dreamhost.com) Quit (Read error: Connection reset by peer)
[5:59] * tren (~Adium@S010658b035f97097.vc.shawcable.net) Quit (Quit: Leaving.)
[5:59] * rturk (~rturk@ps94005.dreamhost.com) Quit (Ping timeout: 480 seconds)
[5:59] * rturk (~rturk@ps94005.dreamhost.com) has joined #ceph
[6:04] * dmick (~dmick@2607:f298:a:607:1a03:73ff:fedd:c856) Quit (Quit: Leaving.)
[6:09] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[6:09] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[6:48] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[6:48] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[6:52] * miroslavk (~miroslavk@c-98-248-210-170.hsd1.ca.comcast.net) has joined #ceph
[6:54] * scuttlemonkey (~scuttlemo@12.239.189.130) Quit (Quit: This computer has gone to sleep)
[7:14] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[7:14] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[7:14] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[7:14] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit ()
[7:22] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[7:31] * miroslavk (~miroslavk@c-98-248-210-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[8:31] * LadyGrim (~LadyGrim@28IAAH5SB.tor-irc.dnsbl.oftc.net) has joined #ceph
[8:32] * LadyGrim (~LadyGrim@28IAAH5SB.tor-irc.dnsbl.oftc.net) Quit ()
[8:44] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[8:44] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[9:04] * EmilienM (~EmilienM@83.167.43.235) has joined #ceph
[9:05] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[9:05] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[9:05] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[9:07] * loicd (~loic@178.20.50.225) has joined #ceph
[9:28] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[9:28] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[9:31] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:40] * deepsa (~deepsa@122.172.44.242) Quit (Ping timeout: 480 seconds)
[9:41] * deepsa (~deepsa@122.167.174.236) has joined #ceph
[9:43] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[9:50] * ecco_ (~ecco@h-208-141.a163.corp.bahnhof.se) has joined #ceph
[9:51] * Leseb (~Leseb@193.172.124.196) has joined #ceph
[9:51] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) has joined #ceph
[9:57] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[9:57] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[9:58] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) Quit (Quit: adjohn)
[10:11] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[10:28] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[10:48] * Cube1 (~cube@12.248.40.138) has joined #ceph
[10:48] * Cube (~cube@12.248.40.138) Quit (Read error: Connection reset by peer)
[10:50] * deepsa_ (~deepsa@115.241.25.164) has joined #ceph
[10:55] * deepsa (~deepsa@122.167.174.236) Quit (Ping timeout: 480 seconds)
[10:55] * deepsa_ is now known as deepsa
[10:56] * ecco_ (~ecco@h-208-141.a163.corp.bahnhof.se) Quit (Quit: This computer has gone to sleep)
[11:02] <jtang> just saw -- https://www.42on.com/events/, are there more details on this event? I'm going to try and send a someone from our offices to this event
[11:09] <joao> so far we know it's going to be on November 2nd in Amsterdam, and that Sage and (I think) Greg are going
[11:09] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[11:10] <joao> I, for one, don't know much more than that :)
[11:10] * gregorg_taf (~Greg@78.155.152.6) has joined #ceph
[11:10] * gregorg (~Greg@78.155.152.6) Quit (Read error: Connection reset by peer)
[11:12] * deepsa_ (~deepsa@122.172.212.203) has joined #ceph
[11:16] * deepsa (~deepsa@115.241.25.164) Quit (Ping timeout: 480 seconds)
[11:16] * deepsa_ is now known as deepsa
[11:18] <wido> jtang: What do you want to know?
[11:29] * edv (~edjv@50-46-210-159.evrt.wa.frontiernet.net) Quit (Quit: Leaving.)
[11:29] * miroslavk (~miroslavk@173-228-38-131.dsl.dynamic.sonic.net) Quit (Quit: Leaving.)
[11:29] * mdxi (~mdxi@74-95-29-182-Atlanta.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[11:42] <jantje> wido: does ceph already do deduplication ?
[11:43] <wido> jantje: No, it doesn't do deduplication
[11:43] <jantje> bummer
[11:43] <jantje> thanks
[11:43] <wido> You meant the POSIX filesystem or RADOS below?
[11:43] * mdxi (~mdxi@74-95-29-182-Atlanta.hfc.comcastbusiness.net) has joined #ceph
[11:44] <jantje> what do you mean
[11:49] <wido> jantje: What I meant, do you want to use the POSIX filesystem?
[11:49] <wido> Or do you want to use RADOS/RBD?
[11:49] <jantje> posix fs
[11:49] <jantje> as in, I just want an ordinary filesystem
[11:50] <wido> Ah, ok. At this point I'd not recommend to use the filesystem since that is still under heavy development and didn't get that much attention lately
[11:53] <jantje> it has been a long time since I looked at ceph
[11:53] <jantje> is there an up to date 'feature' list
[11:55] <jantje> never mind :)
[11:55] <wido> jantje: I guess dutch or belgian?
[11:56] <jantje> .be
[11:57] <wido> You should come to the event November 2nd, useful if you want to catch up
[11:58] <jantje> where's that
[12:00] <joao> Amsterdam
[12:01] <joao> https://www.42on.com/events http://www.inktank.com/news-events/event/ceph-workshops-amsterdam/
[12:01] <jantje> is there already commercial support in .be for ceph
[12:03] <wido> jantje: Yes, from November there will be. 42on will do so
[12:04] <joao> I thought inktank also provided commercial support @ wherever
[12:05] <joao> oh goody, just found a marvelous yaml lib for c++
[12:05] <wido> joao: InkTank does indeed
[12:05] <jantje> That's nice
[12:08] <jantje> What I really need is, I think, is that someone can provide support when someone or somethings fucks up the entire cluster
[12:13] <wido> jantje: That is possible. Although the filesystem can't be supported at the moment since it needs more attention
[12:13] <wido> RBD and RADOS though can
[12:22] <andreask> jantje: companies like hastexo also provide commercial support for rados/rbd in Europe
[12:57] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[12:58] <jtang> wido: since i'm going, im making someelse go i'd like them to get the most out of it
[12:58] <wido> jtang: What are you looking for?
[12:58] <jtang> I'd like to see topics like "best practices", deployment scenarios and use cases
[12:59] <jtang> some administrative tasks that an admin might want to do in a live deployment
[12:59] <jtang> recovery scenarios
[12:59] <jtang> failure scenarios
[12:59] <jtang> discussions on minimal setups to maximise reliability and performance
[12:59] <jtang> things like that
[13:00] <wido> There will be talks about Ceph deployments out there, talking about the challenges
[13:00] <jtang> wido: i meant to say, im not going
[13:00] <jtang> but one of my co-workers might be (if i can get approval for them to go)
[13:00] <wido> It will be a technical day, not a promotional day for commercial things, it's tech talks
[13:00] <jtang> the joys of working in academia and budgets!
[13:01] <wido> recovery is something that hasn't been covered that much, but you can always ask questions
[13:01] <Fruit> academia? just fill out the expenses form :)
[13:01] <jtang> i'm interested in small to medium sized installations
[13:01] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Quit: Leaving.)
[13:02] <jtang> as in 30 to 300tb of usable space
[13:02] <wido> jtang: With RBD? Like OpenStack or CloudStack?
[13:02] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Quit: Leaving.)
[13:04] <jtang> wido: no stacks at all
[13:05] <wido> jtang: The posix filesystem? Or just plain RBD?
[13:05] <jtang> i'm looking more at either the radosgw or the cephfs itself for our application(s)
[13:05] <jtang> it depends on our approach to integrating the storage component with the preservation repository that we are building
[13:05] <wido> Ah, yes. Well, the RGW should be covered. The cephfs as well, but cephfs is still under development
[13:05] <wido> but all aspects of Ceph will be covered with hands-on sessions
[13:06] <jtang> actually that was one of the questions i had
[13:06] <jtang> will the handon sessions be done on a VM ?
[13:06] <jtang> or is it assumed that the participants have ubuntu on their laptop (in our case it will be SL6 if someone goes from here)
[13:07] <wido> jtang: Having a VM for every participant would be a challenge :) But we assume you have your laptop
[13:07] <jtang> wido: heh, look at vagrant ;)
[13:08] <wido> jtang: I know, but having it on that location
[13:08] <jtang> i've been testing in a vagrant vm myself
[13:08] <jtang> its not as automated as i would like, but its good enough for testing and learning
[13:08] <wido> We could always deploy hunderds of VMs with *Stack, but it's more a infra point on that specific location
[13:08] <jtang> its also sl6 as well so its not what you want
[13:16] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) Quit (Ping timeout: 480 seconds)
[13:20] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[13:21] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[13:23] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[13:23] <jtang> will there be a comparison on ceph against other distributed/parallel fs's on the day?
[13:23] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[13:24] <jtang> at least on a feature list (though performance comparisons on the same hardware would also be nice)
[13:25] <wido> jantje: The complete program isn't definitive yet, I have a call about that in 4 hours
[13:25] <wido> It's only one day, so you can't squeeze everything in a single day
[13:25] <jtang> :)
[13:25] <jtang> just trying to find out what might end up on the agenda
[13:26] <jtang> im trying to contain myself before sc12 to ask more of these questions
[13:32] * gucki (~smuxi@84-73-207-147.dclient.hispeed.ch) has joined #ceph
[13:32] <gucki> hi there
[13:33] <gucki> i'm using a 2.6.32 kernel and would like to use rbd for block devices. however the rbd module seems to be missing. where can i get it? :)
[13:33] <gucki> ceph debian package is installed and the test ceph cluster is running fine...i only need the missing kernel module :)
[13:33] * tziOm (~bjornar@194.19.106.242) has joined #ceph
[13:35] <exec> gucki: you can use kernel from backport (3.2), however, you shouldn't use osd and rbd kernel client on the same host
[13:36] <gucki> exec: no, i need to use the openvz kernel. isn't it possible to get the module running with it? :(
[13:36] <gucki> exec: both on same machine are bad because of deadlocks in case of memory pressure?
[13:36] <gucki> exec: just like with nbd?
[13:39] <jtang> gucki: i ran into the same problem you did
[13:39] <jtang> i've a dodgy repo with some reverts at https://github.com/jcftang/ceph-client-standalone/tree/sl6
[13:39] <jtang> it works, but i havent tested it extensively as we just ended up instsalling a mainline kernel on some of our test systems
[13:40] <jtang> ive only tested it on rhel like systems
[13:41] <exec> gucki: yup, deadlocks. not sure about openvz kernel. I'm using qemu under libvirt
[13:44] * jamespage (~jamespage@tobermory.gromper.net) Quit (Quit: Coyote finally caught me)
[13:44] * jamespage (~jamespage@tobermory.gromper.net) has joined #ceph
[13:45] <gucki> jtang: great, i'll try to compile it now. do you think it's stable enough for "heavy" production usage?
[13:46] <gucki> exec: mh, i hoped they elimnated those and that there are only deadlocks left when using an nbd device for swapping. can it even deadlock when not using as a swap device? :(
[13:51] <jtang> gucki: no probably not
[13:51] <jtang> i found the repo from the ceph github account
[13:51] <jtang> and one of the guys in work here dug around why it wasnt building for sl6's kernel
[13:51] <jtang> and i just reverted the commits which caused the build to fail and recorded
[13:52] <jtang> i have no guarantee's that it does what it is supposed to
[13:52] <jtang> it was enough for me to startup cephfs to tune the system for testing purposes
[13:53] <gucki> jtang: mh, compiling doesn't seem to be that easy on debian as some files are missing. i'm also now not sure if it's worth the work, because i really need something stable..
[13:54] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[13:54] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[14:13] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) has joined #ceph
[14:23] <stan_theman> looks like the issue tracker needs some love: http://tracker.newdream.net/
[14:41] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[14:44] <nhm> good morning all
[14:45] <wido> morning :)
[14:51] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[15:04] * aliguori (~anthony@cpe-70-123-130-163.austin.res.rr.com) has joined #ceph
[15:25] * dty (~derek@pool-71-178-175-208.washdc.fios.verizon.net) Quit (Quit: dty)
[15:26] * hk135 (~root@89.30.48.254) has joined #ceph
[15:26] <hk135> hello
[15:38] <hk135> ECHO
[15:38] <hk135> echo
[15:38] * hk135 (~root@89.30.48.254) Quit (Quit: leaving)
[15:46] * dty (~derek@129-2-129-153.wireless.umd.edu) has joined #ceph
[15:52] * scuttlemonkey (~scuttlemo@12.239.189.130) has joined #ceph
[15:58] * rlr219 (43c87e04@ircip1.mibbit.com) has joined #ceph
[16:03] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) has joined #ceph
[16:03] <jtang> gucki: dunno, i haven't been testing on debian as that's not what we have in production here
[16:03] * gaveen (~gaveen@112.134.113.93) has joined #ceph
[16:03] * tziOm (~bjornar@194.19.106.242) Quit (Remote host closed the connection)
[16:05] * loicd (~loic@178.20.50.225) Quit (Ping timeout: 480 seconds)
[16:09] <rlr219> I am confused about the cach settings for a vm using RBD. In my config line I have <source protocol='rbd' name='images/vm:rbd_cache=true:rbd_cache_size=67108864:rbd_cache_max_dirty=25165824'>. I have also tried it with rbd_cache_max_dirty=0. But when I run the VM with the caching enabled, I start seeing Kernel warnings for BUG: soft lockup - CPU#4 stuck for
[16:09] <rlr219> 23s! What is the best settings for caching so that my VMs do not have the cpu pausing?
[16:16] * dty (~derek@129-2-129-153.wireless.umd.edu) Quit (Quit: dty)
[16:21] * loicd (~loic@magenta.dachary.org) has joined #ceph
[16:23] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[16:27] <wido> rlr219: I assume you see those inside the VM?
[16:27] <rlr219> yes
[16:27] <wido> Ah, yes, I should read better
[16:27] <wido> Any special settings? What is running inside the VM? Which kernel?
[16:28] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) Quit (Quit: gminks_)
[16:30] <rlr219> my VMs are all Ubuntu precise. Each Vm is pretty much a single purpose server. The one I am testing with is running cobbler.
[16:30] <rlr219> kernel 3.2.0-24-virtual
[16:30] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) has joined #ceph
[16:30] * cblack101 (c037362a@ircip3.mibbit.com) has joined #ceph
[16:32] <wido> Hmm, that's noting weird
[16:32] <cblack101> Question: I run a 'service ceph restart' on an OSD host and looked at the PIDs with a 'ps ax | grep ceph' before and after the restart... They are the same numbers... Did ceph restart?
[16:34] <cblack101> http://mibpaste.com/3rjiIy Here's the CLI output...
[16:36] <wido> rlr219: I don't have a clue at the moment. Didn't play with the RBD caching that much yet
[16:38] * dty (~derek@testproxy.umiacs.umd.edu) has joined #ceph
[16:44] <rlr219> wido: thanks. I am still playing with the setting. Just wanted to go in right direction
[16:46] * tren (~Adium@S010658b035f97097.vc.shawcable.net) has joined #ceph
[16:46] <wido> rlr219: You might want to check out: http://eu.ceph.com/docs/master/config-cluster/rbd-config-ref/
[16:48] * tren (~Adium@S010658b035f97097.vc.shawcable.net) Quit ()
[16:50] <wido> cblack101: It doesn't seem so
[16:50] <wido> do you have somewith thing "auto start" in the ceph.conf?
[16:50] <wido> Or did you change the hostname lately?
[16:51] * mgalkiewicz (~mgalkiewi@staticline-31-182-149-180.toya.net.pl) has joined #ceph
[16:52] * scuttlemonkey (~scuttlemo@12.239.189.130) Quit (Quit: This computer has gone to sleep)
[16:52] <cblack101> I am running 'service ceph -a restart osd.XX' from mon.1 and that seems to work fine, will use this method to restart OSDs in the future
[16:54] <wido> cblack101: Would you mind sharing your ceph.conf?
[16:58] * Kioob (~kioob@luuna.daevel.fr) has joined #ceph
[17:00] <cblack101> Here's the file: http://mibpaste.com/AstSwY - I know there's probably a less verbose way of doing this, any suggestions to simplify would be quite welcome :-)
[17:00] <slang> stan_theman: fyi, the issue tracker is back
[17:00] <stan_theman> i just noticed!
[17:01] <stan_theman> did that happen a bit ago or should i get a lotto ticket?
[17:02] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:03] <nhm> stan_theman: the tracker is a bit nondeterministic... ;)
[17:04] <stan_theman> :P
[17:04] <slang> stan_theman: I think it was just restarted
[17:11] * Cube1 (~cube@12.248.40.138) Quit (Quit: Leaving.)
[17:11] * Cube (~cube@12.248.40.138) has joined #ceph
[17:20] * tren (~Adium@184.69.73.122) has joined #ceph
[17:25] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:27] * lofejndif (~lsqavnbok@9YYAAJLM4.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:27] * danieagle (~Daniel@177.133.175.73) has joined #ceph
[17:28] * jlogan1 (~Thunderbi@2600:c00:3010:1:4d70:2bbd:6949:8d94) has joined #ceph
[17:30] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) Quit (Quit: gminks_)
[17:32] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) has joined #ceph
[17:38] * scuttlemonkey (~scuttlemo@38.122.20.226) has joined #ceph
[17:43] * gucki (~smuxi@84-73-207-147.dclient.hispeed.ch) Quit (Remote host closed the connection)
[17:45] * Kioob (~kioob@luuna.daevel.fr) Quit (Ping timeout: 480 seconds)
[17:46] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[17:46] * Tv_ (~tv@2607:f298:a:607:5c1e:e9a0:aa30:35e7) has joined #ceph
[17:51] * EmilienM (~EmilienM@83.167.43.235) has left #ceph
[17:56] * scuttlemonkey (~scuttlemo@38.122.20.226) Quit (Quit: This computer has gone to sleep)
[17:59] * scuttlemonkey (~scuttlemo@38.122.20.226) has joined #ceph
[18:04] * mgalkiewicz (~mgalkiewi@staticline-31-182-149-180.toya.net.pl) has left #ceph
[18:08] <jtang> cool, just go approval to send at least one of my co-workers to the ceph workshop next month
[18:09] <jtang> gonna see if i can convince another one of the team to pop along
[18:11] <nhm> jtang: cool!
[18:11] * danieagle (~Daniel@177.133.175.73) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[18:12] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[18:13] * adjohn (~adjohn@108-225-130-229.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[18:17] * Leseb (~Leseb@193.172.124.196) Quit (Quit: Leseb)
[18:24] <tren> Is anyone around? I had a osd node crash, and now I have 2 osd's that won't start
[18:26] * rweeks (~rweeks@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[18:26] * mamalujo (~damjan@85.114.54.27) has joined #ceph
[18:27] <gregaf> hi tren
[18:28] <tren> Morning :)
[18:28] <gregaf> get my email?
[18:28] <tren> I did
[18:28] <tren> unfortunately I had an osd node crash this morning
[18:28] <tren> which has caused the file system to hang
[18:28] <tren> :/
[18:28] <tren> losing 12 osd's out of 192 shouldn't cause the file system to lock :/
[18:29] <gregaf> 2x replication and two lost domains, right?
[18:29] <tren> and now 10 of the 12 are back (2 are crashing on startup)
[18:29] <tren> no
[18:29] <tren> 3x replication
[18:29] <gregaf> ah
[18:29] <gregaf> yeah, not so much then
[18:29] <benpol> I'm curious about the ceph workshop that was mentioned a few moments ago, where can I find more details?
[18:29] <gregaf> benpol: https://www.42on.com/events/
[18:30] <benpol> gregaf: thanks!
[18:30] <gregaf> tren: so what's the story of the OSDs?
[18:31] <tren> gregaf: dunno :/ I'm getting a pastebin together
[18:31] <tren> gregaf: machine locked today, had to kick it via ipmi
[18:31] <gregaf> sorry, I mean observed systems, beyond "crash and now won't reboot" ;)
[18:32] <tren> lol
[18:32] <tren> I'm getting details for ya now
[18:32] <tren> *waits for pastebin to load*
[18:32] <yehudasa> gregaf: you'll be happy to know that there's another branch waiting for your comments: wip-3114
[18:33] <gregaf> k, will do
[18:33] <tren> gregaf: here's the crash from one osd: http://www.pastebin.ca/2239845
[18:34] <gregaf> sjust, help...
[18:34] <sjust> lookin
[18:34] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) Quit (Quit: gminks_)
[18:35] <sjust> it's crashing due to being unable to read one of the pgs
[18:36] <sjust> tren: what is the output of ceph -s
[18:36] <tren> second paste: http://pastebin.ca/2239848
[18:36] <sjust> same problem
[18:36] <sjust> indicates on disk state corruption of some sort
[18:36] <tren> http://pastebin.ca/2239851 <- ceph -s
[18:37] <tren> sjust: makes sense as the machine had to be rebooted via ipmi
[18:37] <sjust> ah...
[18:37] <tren> looks like it's repairing at the moment
[18:37] <sjust> is the 213237 objects degraded number going down?
[18:37] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) has joined #ceph
[18:38] <tren> down to 133k
[18:38] <tren> yup
[18:38] <sjust> ok, let me know what all that stops changing
[18:38] <tren> k
[18:38] <gregaf> tren: is CephFS still stuck?
[18:38] <sjust> with any luck it'll be able to go clean without the two busted osds and you can just recreate them
[18:38] <gregaf> he had 3x replication so it should be all good
[18:38] <tren> gregaf: yes
[18:39] <tren> 78k
[18:39] <gregaf> have you tried mounting a new client?
[18:39] <gregaf> I'm curious if it's the MDS or the client that's stuck
[18:40] <gregaf> and if the problem is that it's not correctly resending requests to new primaries or what
[18:40] <sjust> possibly
[18:40] <sjust> might also just be really slow for the moment
[18:40] <tren> sjust: the mds hasn't switched over since yesterday.
[18:40] <sjust> switched over?
[18:40] <tren> sjust: sorry, failed over
[18:40] <tren> ;)
[18:40] <sjust> ah
[18:41] <tren> none of this today has caused the mds to fail to another node
[18:41] <gregaf> tren: could you try connecting a new client and see if it can do anything on the FS?
[18:41] <sjust> it shouldn't need to
[18:41] <tren> gregaf: should I wait until the repair process is complete?
[18:41] <gregaf> but obviously something's wrong, so let's try and narrow down where the hang is
[18:42] <gregaf> tren: no; all the PGs are active so all data is accessible; things should be going
[18:43] <tren> I was able to mount it on another machine and browse the tree
[18:43] <tren> I can also browse the tree on the client with the stuck rsync process
[18:44] <tren> rsync just started up again
[18:44] <gregaf> hrm
[18:44] <sjust> sounds like the mds was slowly forcing its objects to re-replicated
[18:44] <gregaf> sounds like a missed wakeup or something then
[18:44] <tren> wow
[18:44] <tren> that's creepy
[18:44] <gregaf> sjust: …?
[18:44] <tren> it only runs if I have a strace attached to it
[18:44] <sjust> tren: oh
[18:45] <tren> as soon as I detach the strace from the rsync, it stops again
[18:45] <tren> I've never seen that before. :/
[18:45] <gregaf> umm
[18:45] <sjust> gregaf: just a guess, might have been just suffering from preposterously slow writes due to having to replicated the object each time
[18:45] <gregaf> I've seen strace wake it up before
[18:45] <sjust> *rereplicate
[18:45] <gregaf> I don't think I've seen it turn off on quit like that
[18:45] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[18:45] <gregaf> sjust: ah, I see
[18:46] <sjust> obviously, that explanation no longer seems likely
[18:46] <gregaf> although strace and weird blocking does sound familiar to me
[18:46] <gregaf> sagewk, do you remember the circumstances last time we saw something like that?
[18:47] <nhm> gregaf: attaching/detaching strace seems to have strange effects on things some times.
[18:47] <nhm> gregaf: I've had detaching kill OSDs before.
[18:48] <Tv_> tren: i've seen strace (ptrace) change some sort of timer behavior, even before i ever dealt with ceph
[18:50] <tren> Tv_: Hmm…it's strange…but kinda fascinating
[18:50] <Tv_> yeah strace will essentially deliver spurious wakeups
[18:50] <tren> But peripheral to this issue
[18:50] <tren> so how do I go about bringing my 2 dead osd's back to life?
[18:51] <Tv_> so if there was a timer handling bug in the client code, strace could hide the bug
[18:51] <Tv_> oh also strace will make EINTR happen a lot more
[18:51] <tren> Tv_: laymans term? :)
[18:52] <Tv_> tren: interrupted system calls that the library or app is supposed to retry
[18:52] <gregaf> tren: well, you killed their data stores (can you tell how exactly from that backtrace sjust?) with the powered reset, so you'll need to wipe them and let them recover; sjust can walk you through that faster than I can
[18:53] <tren> gregaf: thank you :)
[18:53] <sjust> tren: the crash was in the bit where the osd reads its state on startup, they are at least a bit corrupted
[18:53] <sjust> if we can get your osd cluster healthy without them, it's easiest to kill them and re create them
[18:53] <tren> sjust: the cluster is healthy
[18:54] <tren> ocr35-ire protus # ceph -s
[18:54] <tren> health HEALTH_OK
[18:54] <tren> monmap e1: 3 mons at {fern=10.87.1.88:6789/0,ocr46=10.87.1.104:6789/0,sap=10.87.1.87:6789/0}, election epoch 72, quorum 0,1,2 fern,ocr46,sap
[18:54] <tren> osdmap e161: 192 osds: 190 up, 190 in
[18:54] <tren> pgmap v512263: 74112 pgs: 74112 active+clean; 1689 GB data, 7112 GB used, 276 TB / 283 TB avail
[18:54] <tren> mdsmap e44: 1/1/1 up {0=ocr46=up:active}, 2 up:standby
[18:54] <sjust> basically, you need to wipe the 2 dead osds and re create them
[18:54] <tren> is there a page that goes through how to re-create a dead osd?
[18:54] <sjust> looking it up myself, one sec
[18:54] <tren> I've only made osd's via mkcephfs thus far
[18:54] <tren> thanks :)
[18:56] <nhm> sjust: sounds like a good usability improvement for when we are not swamped...
[18:58] <gregaf> tren: this is actually documented! http://ceph.com/docs/master/cluster-ops/add-or-rm-osds/
[18:58] * mamalujo (~damjan@85.114.54.27) Quit (Ping timeout: 480 seconds)
[18:58] <gregaf> do the manual remove (the parts that are applicable, anyway) and then the manual add
[18:58] <Tv_> nhm: it would have been a lot simpler with the new deploy stuff...
[18:59] <Tv_> nhm: this is why i wish to kill mkcephfs
[18:59] <tren> these steps seem to be for ext4. Do I need to mount xfs with user_xattr?
[18:59] <tren> because I haven't been :/
[18:59] <Tv_> tren: nobody needs user_xattr anymore
[18:59] <tren> okay :)
[18:59] <nhm> Tv_: I should probably sit down and learn how the new deploy stuff works. I haven't looked at it at all. :/
[19:00] <tren> gimme a sec to wipe these osd's and re-add them
[19:00] <Tv_> nhm: still low on docs
[19:00] <tren> nhm: I'd kill for a puppet manifest for deployment.
[19:00] <nhm> tren: A couple of people have expressed interest in getting something going. Not sure what the current state of things is.
[19:01] <Tv_> tren: there is a community-maintained puppet manifest; i can't vouch for state, quality etc
[19:02] <Tv_> ceph-osd(ceph/10): journal not present, not starting yet.
[19:03] <Tv_> whee
[19:03] <tren> Tv: It's not complete. Only handles mon's.
[19:03] <Tv_> external journal hotplug support is starting to work right
[19:03] <Tv_> tren: well, anyone who wants to work on it, i'll happily provide braindumps of what needs to be done
[19:03] <Tv_> but we're gonna cover juju before puppet
[19:04] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:05] <tren> do I have to do the osd authentication key if I'm not using cephx auth?
[19:05] <Tv_> tren: nope
[19:06] <tren> on the initializing osd directory I'm getting an error
[19:06] <tren> ocr35-ire ceph # ceph-osd -i 48 --mkfs --mkkey
[19:06] <tren> 2012-10-05 10:05:54.476996 7fb8b3da9780 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-48: (2) No such file or directory
[19:07] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[19:07] <gregaf> does the directory exist?
[19:07] <tren> that's not where it's supposed to go :/
[19:07] <tren> my osd's go in /var/ceph
[19:07] <tren> do I need to pass -c /etc/ceph/ceph.conf?
[19:07] <Tv_> tren: that's the default config file
[19:07] <gregaf> do you have a ceph.conf file on the node?
[19:07] <tren> got it
[19:08] <tren> needed to use 048
[19:08] <tren> :)
[19:08] <Tv_> huh what
[19:08] <gregaf> does that contain something which will tell the new osd where to put its data? (either in the OSD section with metavariable expansion or in the specific one)
[19:08] <tren> tv_: I'm a little anal with naming. I have 192 osd's. So I 0 pad the ones under 100
[19:08] <Tv_> tren: you probably really shouldn't be zero-padding your osd names
[19:08] <tren> Tv_: Why?
[19:09] <Tv_> tren: because things might assume str(int(name)) == name
[19:09] <Tv_> tren: osds are a special case; the name isn't really a string, it's a small integer
[19:09] <Tv_> tren: for everything else, the name really is a string
[19:09] <gregaf> we're still stupid about OSD IDs
[19:09] <gregaf> but, I think it'll work in this case
[19:10] <tren> Tv_: It's worked perfectly in practice…and there's no warnings anywhere about naming conventions for osd's
[19:11] <Tv_> *shrug* i intend to make humans not care about the numbers anyway, so i win the end game either way ;)
[19:12] <tren> :D
[19:13] <tren> one question. I did a step I didn't need to do as I had the osd already registered. so it made osd.192
[19:13] * Ryan_Lane (~Adium@216.38.130.165) has joined #ceph
[19:13] <tren> but there isn't supposed to be an osd.192
[19:13] <tren> how do I remove that from the osd tree?
[19:13] <tren> nm
[19:13] <Tv_> tren: see the shrinking steps on the same docs page
[19:13] <tren> if I scroll further down the page I see the answer
[19:13] <tren> lol
[19:13] <tren> sorry :/
[19:13] <tren> I blame lack of coffee
[19:15] <tren> the steps don't seem to be working :/
[19:15] <tren> and my osd tree is looking messed up
[19:15] <tren> it's taken osd.0 from the host it's on and now it's floating
[19:15] <tren> and osd.192 won't remove
[19:17] * Kioob (~kioob@luuna.daevel.fr) has joined #ceph
[19:17] <joao> this is *so* cool
[19:18] <joao> the tracker shows the revisions associated with a given issue
[19:18] <gregaf> tren: ahhh, either you typoed or the docs don't match the version you're on
[19:18] <gregaf> (but I thought the version you were on was fixed up so this wasn't so fragile; maybe it's in master but not .52)
[19:18] <tren> gregaf: I'm putting my osd tree in a pastebin.
[19:19] <gregaf> can you pastebin the commands you ran as well?
[19:19] <tren> http://pastebin.ca/2239896
[19:19] <gregaf> bbiab, standup is starting
[19:19] * Kioob (~kioob@luuna.daevel.fr) Quit ()
[19:20] * gminks_ (~ginaminks@67-198-111-109.dyn.grandenetworks.net) Quit (Quit: gminks_)
[19:20] <gregaf> for the moment, I believe you want to run "ceph osd crush set 0 osd.0 1 host=ocr31-ire"
[19:20] <gregaf> to put it back
[19:21] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[19:22] <tren> http://pastebin.ca/2239898
[19:22] <tren> That fixed it
[19:22] <tren> Thank you :)
[19:29] * The_Bishop (~bishop@2001:470:50b6:0:b02f:c7c:6a8e:eba6) has joined #ceph
[19:43] <tren> wow…I'm impressed. The cluster has fully healed
[19:43] <tren> the 2 osd's I had to wipe are back to where they were at 7am. Very nice!
[19:44] * dmick (~dmick@2607:f298:a:607:1a03:73ff:fedd:c856) has joined #ceph
[19:47] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[19:48] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit ()
[19:49] * BManojlovic (~steki@212.200.240.160) has joined #ceph
[19:49] <tren> gregaf: you around? I have a few questions about the email you sent me. Now that the cluster is stable again I can get you the info you asked for
[19:49] <gregaf> yeah
[19:50] <tren> am I passing those options to the active mds?
[19:50] <gregaf> no, for ceph-fuse
[19:50] <tren> how do I do that?
[19:50] <gregaf> it looks like the problem is not with the metadata server but with the client
[19:50] <gregaf> you can put those options in the client's config
[19:50] <tren> I don't have a client config.
[19:50] * Beachtoberg (~Beachtobe@196-209-240-119.dynamic.isadsl.co.za) has joined #ceph
[19:50] <tren> I just have my global ceph.conf
[19:51] <gregaf> or pass them as command-line arguments (--debug_client 20 etc)
[19:51] <tren> so I'll need to stop the rsync, unmount the fs, remount it with those options, and restart the rsync?
[19:51] * miroslavk (~miroslavk@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[19:52] <gregaf> yeah; if you can do so that'd be best
[19:52] <gregaf> is your client creating a log yet, do you know?
[19:52] <gregaf> if not we'll need to specify a log location too
[19:52] <gregaf> --log-file /path/to/log
[19:53] <tren> client.admin.log?
[19:53] <gregaf> possibly — what are the contents?
[19:53] <tren> I see ceph-fuse in there
[19:53] <tren> though nothing's been written to it in a while
[19:54] <gregaf> okay, that's the one then, yes
[19:54] <tren> k
[19:54] <gregaf> just be aware it's going to get large pretty quickly, so if it's on a small partition you'll want to specify a different location
[19:55] * BManojlovic (~steki@212.200.240.160) Quit (Quit: Ja odoh a vi sta 'ocete...)
[19:55] * Beachtoberg (~Beachtobe@196-209-240-119.dynamic.isadsl.co.za) Quit (Quit: Leaving)
[19:55] <tren> k, I made a temp lv for it
[19:57] <tren> gregaf: sanity check ceph-fuse --log-file /log/ceph-fuse.ocr31-ire.log --debug_client 20 --debug_ms 1 -m 10.87.1.87:6789,10.87.1.88:6789,10.87.1.104:6789 /data1/ceph
[19:57] <tren> that look good?
[19:58] <gregaf> yep!
[19:59] <tren> k, it's running
[19:59] <tren> how long do you want me to run this?
[20:00] <tren> 43mb already
[20:00] <gregaf> probably for some files to finish rsyncing is long enough, but the longer the better given my failure to guess accurately so far ;)
[20:01] <tren> it's not going to rsync any files :/
[20:01] <tren> it has to get like 1.6TB deep into the tree before that happens
[20:01] <tren> here, let me rsync to an alternate location
[20:01] <gregaf> can you just rsync a subdir or something then?
[20:01] <tren> There
[20:01] <tren> :)
[20:02] <gregaf> sorry, a meeting I thought was cancelled just called me; I'll be back later
[20:02] <tren> have it rsyncing to another place
[20:02] <tren> okie
[20:02] <tren> you will have logs
[20:02] <dmick> There Will Be Logs
[20:02] <tren> :D
[20:03] * MikeMcClurg (~mike@firewall.ctxuk.citrix.com) Quit (Quit: Leaving.)
[20:05] * gminks_ (~ginaminks@108-210-41-138.lightspeed.austtx.sbcglobal.net) has joined #ceph
[20:07] <rweeks> Logs, Logs, they're big, they're heavy, they're wood
[20:08] * maelfius (~mdrnstm@66.209.104.107) has joined #ceph
[20:08] <tren> I think I'd need a forest to print out these logs. I guess there'd be a certain type of completeness to do that ;)
[20:08] <gregaf> mmm, deliciously short meetings that provide information
[20:09] <tren> gregaf: those are the best
[20:09] <gregaf> indeed
[20:09] <tren> gregaf: got about 900MB of logs now
[20:09] <tren> gregaf: think that's enough?
[20:09] <gregaf> well, actually, meetings are usually a lame way of conveying information, but in this case it worked out well since it was really only 3 of us so it was more like grabbing somebody to ask questions about undocumented stuff ;)
[20:09] <gregaf> tren: has it completed rsyncing several files?
[20:10] <tren> gregaf: it's rsynced almost 1gb of data
[20:10] <tren> gregaf: and I have almost 1gb of logs
[20:10] <gregaf> heh
[20:10] <tren> memory use is at 104mb
[20:11] <gregaf> the files are pretty small, right?
[20:11] <gregaf> (ie, that's like 10-1000 files that have finished, not just one big file)
[20:11] <tren> yup
[20:11] <tren> lots and lots of small files
[20:11] <tren> tif's actually
[20:11] <gregaf> my suspicion is it's good then
[20:12] <gregaf> what are they, like 4x4 pixels? :p
[20:12] <tren> they're 60k to 1.5mb ish
[20:12] <tren> no, faxes
[20:12] <gregaf> I kid — I just remember TIFF being really large
[20:12] <tren> okay, I'll stop the rsync and unmount the fs and get you the logs
[20:14] <tren> gregaf: when I stopped the rsync, and unmounted the fs, the ceph-fuse process is still sticking around, and it's growing
[20:14] <tren> gregaf: it's up to 700ish mb
[20:15] <tren> wow, and the logs are growing fast too
[20:15] <Tv_> sagewk: just pushed three lost only-in-stable commits to master
[20:15] <Tv_> sagewk: git cherry master github/stable|sed -n 's/^\+ //p'|xargs git show
[20:16] <gregaf> tren: this might actually have the same root cause; if there are a bunch of messages piling up because it's trying to release stuff and finding it can't do so
[20:16] <Tv_> sagewk: that lists what's in stable but not in master; the three commits from me are all false positives, they are in master, just look a little bit different due to context changing
[20:16] <tren> gregaf: should I kill the ceph-fuse process? these logs are going crazy
[20:16] <tren> 4.9gb
[20:16] <gregaf> go for it — if there's anything there we already have it
[20:20] <tren> gregaf: okay, I'm bzipping the file now. I'll toss it on ftp for you
[20:25] * JT (~john@astound-69-42-3-170.ca.astound.net) has joined #ceph
[20:25] <sjust> JT: no<x> means that the mons will not automatically mark anyone <x>
[20:25] <sjust> so with noin set, the monitors won't mark a newly health osd back in
[20:25] <JT> So noup means it won't mark an OSD up?
[20:26] <sjust> yeah
[20:26] <JT> That's intuitive enough from the naming, but what's the use case? Why would I want to do this?
[20:27] <tren> JT: if a server has crashed and you want to be sure that the osd's on it are healthy
[20:27] <sjust> one example if if you are in the midst of a situation where the monitors are for some reason marking osds up and down repeatedly because of some kind of edge case in the failure detection, you can stop the process and get the cluster back to a stable state
[20:27] <sjust> so as a panic button
[20:28] <JT> I looked into some release notes, and it had mentioned using them in upgrades... e.g., set an osd to nodown noout so that you can upgrade. Is that so it doesn't redistribute data?
[20:28] <sjust> yeah
[20:29] <sjust> we might want a finer grained control in the future
[20:29] <sjust> or better named control perhaps
[20:29] <JT> tren: So the server crashes... what would you do? Set an osd to nodown? So it appears up, even though the server crashed? Noup? So you can ensure they are running before you put them back online?
[20:30] <JT> So these should be used effectively as temporary states then, correct?
[20:30] <tren> JT: I panic ;)
[20:30] <sjust> yes
[20:30] <tren> JT: Actually currently I don't have ceph in the boot process so I can bring things up myself
[20:30] <tren> JT: I actually don't use the noout/noin stuff
[20:31] <sjust> tren: yeah, that would be a pretty blunt hammer
[20:31] <sjust> ...I guess hammers are usually pretty blunt
[20:31] <tren> sjust: I also treat drbd the same way. I always get fearful of clustered storage coming back in and corrupting other parts of it.
[20:32] <sjust> ah
[20:32] <sjust> the case where a server crashes should be pretty safe for ceph
[20:32] <JT> So let me get a little more pointed... if an osd is down, and you set it to nodown, does that mean that clients still try to write data to the osd, or does it only mean that CRUSH doesn't try to rebalance?
[20:32] <tren> sjust: Is there a better way to handle this case?
[20:32] <sjust> it means that the clients will still try to talk to the osd as well
[20:32] <sjust> tren: yeah, just let the failure detection run it's course
[20:33] <sjust> *its
[20:33] <sjust> or if it's coming right up, you can set noout to prevent redistribution
[20:33] <sjust> nodown is usually not a good idea since it'll cause data availability problems (i.e., clients will try to talk to down osd)
[20:33] <JT> Ok. So if I'm upgrading, and I set the osd to noup, clients will NOT try to write data to the OSD, but CRUSH will NOT try to rebalance?
[20:34] <sjust> noup is more useful if you want the monitors to ignore slow osds for a while
[20:34] <sjust> you might set noout to prevent data redistribution
[20:34] <sjust> nodown would cause clients to continue to try to talk to the upgrading osds, so you probably don't want that
[20:35] <JT> When would I want to use noin?
[20:35] <sjust> same as noup
[20:35] <sjust> sorry, it doesn't do the same thing
[20:36] <JT> noin, if the osd is slow and we want to ignore it for awhile?
[20:36] <sjust> but would probably be useful in similar circumstances, the monitors are marking osds down and up, and you want to prevent that from causing continuous redistriution
[20:36] <sjust> yeah
[20:36] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[20:36] <JT> Ok. I'll take a crack at it and get back to you with questions... thanks...
[20:36] <sjust> we have observed cases there the monitors will incorrectly detect osds as failed due to load and mark them down
[20:37] <sjust> this causes other osds to have to work to deal with the new map
[20:37] <sjust> making them slow
[20:37] <sjust> and incorrectly marked out
[20:37] <sjust> *marked down
[20:37] <JT> ah... ok... interesting.
[20:37] <JT> That's due to network latency...overload?
[20:37] <sjust> so you can stop the run away train by setting noup and noin
[20:38] <sjust> usually, it's because something has sent the machine into swap and it can't set heartbeats fast enough
[20:38] <sjust> we have provisions now in the monitor that should notice the pattern and automatically stop it
[20:38] <sjust> but noin/noup can still function as a panic button
[20:39] <Tv_> hehe.. the ceph partition type uuids end in ..ceff2be for in-progress, ..ceff05d for osd data, ..ceff106 for journal ("log")
[20:39] <sjust> heh
[20:39] <dmick> gee, it tastes both bitter and sweet
[20:40] <Tv_> nhm: hey how meaningful is having journal as separate partition on same disk, as opposed to file inside $osd_data_dir ?
[20:40] <Tv_> nhm: should i make partition on same disk be the new default?
[20:41] <Tv_> note: harder to resize
[20:42] <Tv_> i know kernel people have said swap-in-file vs swap-on-partition is not a meaningful difference
[20:42] <Tv_> (assuming non-sparse, contiguous)
[20:45] <nhm> Tv_: I haven't tested jouranl-on-filesystem in really high performance configurations yet. At lower performance levels it doesn't seem to matter much.
[20:46] <nhm> Tv_: all of my high performance tests have been with journal on a partition.
[20:46] <Tv_> nhm: what i feared -- no facts to go by. i guess that means i shouldn't change the default, for now ;)
[20:47] <Tv_> nhm: thanks!
[20:47] <nhm> Tv_: no problem!
[20:50] <tren> gregaf: I'm emailing you the auth details to our ftp
[20:52] <nhm> Tv_: is there a mechanism to allow you to easily provision journals on SSDs? That may be more important.
[20:53] * Cube1 (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[20:55] <nhm> s/SSDs/alternate disks
[21:00] <tren> What was the link to the ceph event that's going on soon?
[21:02] <benpol> tren: https://www.42on.com/events/ (wish I could afford the ticket to Amsterdam!)
[21:03] <tren> benpol: Thanks! I doubt I'll get to go, but the boss didn't say "no" :)
[21:05] <tren> gregaf: Heading to lunch. I emailed you auth info for ftp to the logs :)
[21:07] * rlr219 (43c87e04@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[21:12] * Cube1 (~Adium@12.248.40.138) has joined #ceph
[21:37] * gaveen (~gaveen@112.134.113.93) Quit (Remote host closed the connection)
[21:41] * jstrunk (~quassel@146.6.139.110) has joined #ceph
[21:42] <jstrunk> What filesystems are recommended for use on an SSD journal device?
[21:43] * gminks_ (~ginaminks@108-210-41-138.lightspeed.austtx.sbcglobal.net) Quit (Quit: gminks_)
[21:44] * gminks_ (~ginaminks@108-210-41-138.lightspeed.austtx.sbcglobal.net) has joined #ceph
[21:45] * gminks_ (~ginaminks@108-210-41-138.lightspeed.austtx.sbcglobal.net) Quit ()
[21:49] <joshd> jstrunk: no filesystem - the osd uses the raw device
[21:51] <jstrunk> joshd: this must be a documentation bug: http://ceph.com/docs/master/config-cluster/osd-config-ref/ See "osd journal".
[21:51] <joshd> yeah, thanks for pointing that out
[21:53] <jstrunk> so, I should partition the SSDs for all the OSDs running on each server?
[21:53] * amatter (~amatter@209.63.136.133) Quit (Ping timeout: 480 seconds)
[21:53] <joshd> generally yes
[21:54] <jstrunk> thank you
[21:54] <joshd> you're welcome :)
[22:02] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[22:07] <jefferai> Do any of you run lots of VMs with RBD as the VM backing stores?
[22:07] <jefferai> (obviously, with qemu-rbd)
[22:09] <jefferai> I'm curious as to whether performance is acceptable...also whether it's better to use qemu-rbd or export the rbd with aoe or iscsi
[22:09] <tren> jefferai: I've exported rbd via iscsi
[22:09] <tren> jefferai: it works :)
[22:10] <jefferai> heh
[22:10] <jefferai> tren: how many VMs?
[22:10] <jefferai> how's the performance been?
[22:11] <tren> jefferai: I just did it as a proof of concept. I was able to saturate gigabit for send and receive
[22:11] <jefferai> hm, okay
[22:11] <jefferai> nice to know
[22:11] <tren> jefferai: that was using 56 osd's
[22:11] <tren> jefferai: I used scst
[22:11] <Tv_> nhm: fyi you may enjoy this: http://utcc.utoronto.ca/~cks/space/blog/linux/BlktraceNotes
[22:12] <jefferai> scst?
[22:12] <jefferai> oh, you mean for iscsi
[22:19] <Tv_> nhm: goal is any partition
[22:19] <Tv_> nhm: but i was talking about what to do when user doesn't specify
[22:19] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:19] <Tv_> jefferai: i know some admins with a mighty big setup..
[22:20] <Tv_> jefferai: qemu using librbd is what we recommend; anything else has the feel of a workaround
[22:20] <jefferai> Tv_: okay
[22:24] <tren> Tv_: Is the librbd implementation userland? So you can use on osd nodes?
[22:24] <tren> Tv_: as opposed to the kernel rbd client
[22:25] <Tv_> tren: yup
[22:25] <tren> Tv_: Thanks for confirming that :)
[22:30] * dty (~derek@testproxy.umiacs.umd.edu) Quit (Quit: dty)
[22:43] * alphe (~Santiago@200.111.172.138) has joined #ceph
[22:43] <alphe> hello all I have a strange bug
[22:44] <alphe> os arch linux ceph 0.52 I don't get the right size of the ceph drive after mounting it I see 176GB as size instead of 44TB
[22:45] <Tv_> alphe: how many servers, disks, one osd per disk or something else?
[22:45] <alphe> the strange thing is that another linux ubuntu brand sees it right ...
[22:46] <alphe> 4 servers and 24 disks
[22:46] <Tv_> oh i wonder if that's a 32-bitness bug or something
[22:46] <Tv_> if another box sees it right
[22:47] * davidz1 (~Adium@2607:f298:a:607:64ee:4859:aab8:92e8) has joined #ceph
[22:48] <alphe> os is an arch 64 bits in a VM
[22:49] * miroslavk (~miroslavk@c-98-234-186-68.hsd1.ca.comcast.net) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * The_Bishop (~bishop@2001:470:50b6:0:b02f:c7c:6a8e:eba6) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * chutzpah (~chutz@199.21.234.7) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * Tv_ (~tv@2607:f298:a:607:5c1e:e9a0:aa30:35e7) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * loicd (~loic@magenta.dachary.org) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * SvenDowideit (~SvenDowid@203-206-171-38.perm.iinet.net.au) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * johnl (~johnl@2a02:1348:14c:1720:edf2:aa8:a67e:4d3b) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * nwl (~levine@atticus.yoyo.org) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * ogelbukh (~weechat@nat3.4c.ru) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * benner (~benner@193.200.124.63) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * scheuk (~scheuk@67.110.32.249.ptr.us.xo.net) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) Quit (reticulum.oftc.net charon.oftc.net)
[22:49] * miroslavk (~miroslavk@c-98-234-186-68.hsd1.ca.comcast.net) has joined #ceph
[22:49] * The_Bishop (~bishop@2001:470:50b6:0:b02f:c7c:6a8e:eba6) has joined #ceph
[22:49] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[22:49] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[22:49] * Tv_ (~tv@2607:f298:a:607:5c1e:e9a0:aa30:35e7) has joined #ceph
[22:49] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[22:49] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:49] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[22:49] * SvenDowideit (~SvenDowid@203-206-171-38.perm.iinet.net.au) has joined #ceph
[22:49] * johnl (~johnl@2a02:1348:14c:1720:edf2:aa8:a67e:4d3b) has joined #ceph
[22:49] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) has joined #ceph
[22:49] * scheuk (~scheuk@67.110.32.249.ptr.us.xo.net) has joined #ceph
[22:49] * benner (~benner@193.200.124.63) has joined #ceph
[22:49] * ogelbukh (~weechat@nat3.4c.ru) has joined #ceph
[22:49] * nwl (~levine@atticus.yoyo.org) has joined #ceph
[22:50] <alphe> Tv_ when I tryed to compile ceph in arch 32 bits I had a zillion warning and errors so I recalled that it wass better in 64 bits
[22:50] <alphe> but the os below the vm is a 32bits
[22:50] <iggy> tren: librbd also has caching which the kernel bits don't (yet?)
[22:52] <tren> iggy: That's good to know, and would explain why I could never get the caching to work with the kernel client :)
[22:53] * JT (~john@astound-69-42-3-170.ca.astound.net) Quit (Quit: Leaving)
[22:53] <dmick> alphe: I just fixed a bug in the 32-bit packages; I would expect the build should be workable, at least on Debian and friends
[23:10] <jefferai> Tv_: thanks a bunch for the advice, btw
[23:10] <Tv_> happy to help
[23:31] <alphe> is archlinux a "friend" ?
[23:31] <alphe> hahaha kidding ok but 64 bits isn't better ?
[23:32] <tren> Does ceph require a 64bit host? or it just prefers it?
[23:32] <alphe> as far as I know 64 bits is preferable (it was recommanded with the early version of ceph )
[23:33] <alphe> I will do more tests see if that bug exists with other versions ... it's not big problem for the moment it works great
[23:34] <alphe> around 100Mbps to transfer my files to the ceph cluster :)
[23:34] <alphe> ok I will comeback with better informations I just wanted to know if my problem sounded like a known bug to you
[23:35] <alphe> bye
[23:35] * alphe (~Santiago@200.111.172.138) Quit (Quit: Leaving)
[23:42] <Tv_> tren: ceph-fuse has issues on 32-bit due to small inodes, apart from that it's just that you often want >4GB RAM
[23:42] <Tv_> personally, i haven't run a 32-bit box in ages
[23:42] <tren> Tv_: You can get servers with less than 4GB ram now? :|
[23:42] <tren> ;)
[23:42] <Tv_> tren: my point.. modern computers want to be 64-bit
[23:43] <dmick> laptops still sometimes cheese on the proc, but that's getting less and less common
[23:43] <dmick> not like a system deployment, but it's useful for the developer/experimenter side of things
[23:43] <tren> Tv_: Sorry Tv_…just remembering the old days of 16mb of ram in a 486 server...
[23:43] <dmick> that's "millibits"?
[23:43] <dmick> ;)
[23:43] <Tv_> tren: my original 486 had 4MB RAM and was considered a CAD workstation...
[23:43] * The_Bishop (~bishop@2001:470:50b6:0:b02f:c7c:6a8e:eba6) Quit (Ping timeout: 480 seconds)
[23:44] <tren> Tv_: nice! 1996?
[23:44] <Tv_> i think 93ish
[23:44] <Tv_> actually, 91 or 92
[23:44] <Tv_> as i had it before i got into linux
[23:44] <tren> heh :) it's insane how far things have come. The new servers we get have more memory now than my laptop has storage
[23:45] <dmick> "why back in my day, we used to have to write our bits by longhand, with a quill pen"
[23:45] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:45] <Tv_> my home desktop is 64GB RAM, 10TB 7200RPM disk + SSD
[23:45] * loicd (~loic@82.235.173.177) has joined #ceph
[23:45] <Tv_> it's ridiculous how fast computers are these days
[23:45] <tren> 10TB? :P
[23:45] <Tv_> i had them laying around ;)
[23:46] <tren> nice :)
[23:46] <tren> I'm still rocking an old i7 920 for my home server
[23:47] * scuttlemonkey (~scuttlemo@38.122.20.226) Quit (Quit: This computer has gone to sleep)
[23:47] <rweeks> and dmick, we walked uphill to school! Both ways! In the snow.
[23:47] <Tv_> dmick: my university used to use colorful punch cards for seating arrangements (to ensure two people taking the same exam didn't sit too close to each other)
[23:48] <tren> rweeks: didn't we also have to wrap barbed wire around our feet to give us traction on the ice we had to walk up to get to school?
[23:48] <rweeks> maybe you fortunate ones that HAD barbed wire available.
[23:49] <tren> ha! :D
[23:50] <dmick> luxury!
[23:50] <dmick> we had to make our own barbed wire! out of oatmeal!
[23:52] * The_Bishop (~bishop@2001:470:50b6:0:35fc:e9aa:3704:5f72) has joined #ceph
[23:53] <rweeks> we didn't have oatmeal. All we had was dirty water left over from the laundry.
[23:55] <Tv_> http://www.youtube.com/watch?v=Xe1a1wHxTyo
[23:57] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.