#ceph IRC Log

Index

IRC Log for 2010-08-05

Timestamps are in GMT/BST.

[0:15] <darkfader> wow damn :(
[0:29] * allsystemsarego (~allsystem@188.26.32.97) Quit (Quit: Leaving)
[1:40] * alexxy (~alexxy@79.173.82.178) has joined #ceph
[3:54] * pcish (7a742213@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[5:32] * darkfade1 (~floh@host-82-135-62-109.customer.m-online.net) has joined #ceph
[5:32] * darkfader (~floh@host-82-135-62-109.customer.m-online.net) Quit (Read error: Connection reset by peer)
[6:25] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) has joined #ceph
[6:59] * f4m8_ is now known as f4m8
[7:09] * akhurana is now known as Guest910
[7:09] * Guest910 (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) Quit (Read error: Connection reset by peer)
[7:09] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) has joined #ceph
[7:48] * mtg (~mtg@vollkornmail.dbk-nb.de) has joined #ceph
[7:54] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) Quit (Quit: Osso)
[8:01] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) Quit (Quit: akhurana)
[8:11] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Read error: No route to host)
[8:30] * lidongyang (~lidongyan@222.126.194.154) has joined #ceph
[8:32] * lidongyang (~lidongyan@222.126.194.154) Quit (Remote host closed the connection)
[8:33] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[8:47] * allsystemsarego (~allsystem@188.26.32.97) has joined #ceph
[9:00] * mtg (~mtg@vollkornmail.dbk-nb.de) Quit (Read error: Connection reset by peer)
[9:08] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) has joined #ceph
[9:16] * mtg (~mtg@vollkornmail.dbk-nb.de) has joined #ceph
[9:30] * lidongyang (~lidongyan@222.126.194.154) has joined #ceph
[9:32] * akhurana is now known as Guest920
[9:32] * Guest920 (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) Quit (Read error: Connection reset by peer)
[9:32] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) has joined #ceph
[9:56] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) Quit (Quit: akhurana)
[9:58] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) has joined #ceph
[10:09] * akhurana (~ak2@c-98-232-30-233.hsd1.wa.comcast.net) Quit (Quit: akhurana)
[11:04] <jantje> sagewk: mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs WORKS , while being in /etc/ceph/ mkcephfs -c ceph.conf --allhosts --mkbtrfs does -not- work. So you might want to make sure it's an absolute path (add it to the manual?)
[11:08] <jantje> And doing local ceph mounts to the local MDS does not work (possibily intentional)
[11:12] <jantje> I get mount error 5 = Input/output error all the time :-)
[11:12] <jantje> (It's mounted...)
[13:29] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[13:35] * MarkN (~nathan@59.167.240.178) has joined #ceph
[13:36] <jantje> 10.08.05_13:36:20.670397 7f51be68c710 -- 138.203.10.98:6800/3260 >> 138.203.10.99:6801/3635 pipe(0xbdd050 sd=12 pgs=0 cs=0 l=0).connect claims to be 138.203.10.99:6801/3812 not 138.203.10.99:6801/3635 - wrong node!
[13:46] <darkfade1> is your mds/mon config ok? i'd go check it after that message
[14:03] <jantje> Yea, it should be OK, I only changed my crush map and my decreased the number of OSD's, i did a new mkcephfs and (re)started ceph
[14:04] <jantje> I'm running mds, osd and mon on the same server (3 servers, 3 osd's 3mon, 3 mds)
[14:13] <jantje> found it
[14:13] <jantje> somehow I ended up with a /etc/ceph.conf on my other machines
[14:13] <jantje> and when using init.d/ceph -a , those are used in /etc/ instead of those in /etc/ I think
[14:13] <jantje> or something like that
[14:16] <jantje> mount: 138.203.10.100:/: can't read superblock
[14:16] <jantje> wicked :)
[14:31] <jantje> I really need more information on CRUSH, i'm trying to get a crush map without replication
[14:32] <jantje> rule rbd is having a type replicated, are there other values?
[14:37] <jantje> there's something wrong with my crush map I guess
[14:47] <jantje> It doesn't work with: crushtool --build --num_osds 3 -o file host straw 1 rack straw 1 root straw 3 ; mkcephfs -c /etc/ceph/ceph.conf --allhosts --mkbtrfs --crushmap file , ... start ceph
[14:53] <jantje> damn, I should stop copy pasting stuff from things I'm trying,I always copy paste the wrong stuff
[14:53] <jantje> host: 1 rack:0 root: 0 ofcourse
[14:53] <jantje> (still doenst work :P)
[15:03] <darkfade1> jantje: a few days ago i prepared a new osd vm and when it came to changing the crushmaps i gave up for the day
[15:03] <darkfade1> i hope you'll get it to work but can't help :)
[15:30] <jantje> I got it working by adding the metadata/casdata/rbd rules
[15:30] <jantje> It's all very new, so I really dont have a clear view on what all that means
[15:31] <jantje> Anyway, I have 3 servers with 6 disks total, every disk should get me ~50-70 MB/sec, so I want total transfer rates or at least 200MB/s :P
[15:44] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) Quit (Ping timeout: 480 seconds)
[15:46] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) has joined #ceph
[15:47] * f4m8 is now known as f4m8_
[16:03] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) has joined #ceph
[16:06] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[16:07] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) Quit ()
[16:07] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) has joined #ceph
[16:31] * mtg (~mtg@vollkornmail.dbk-nb.de) Quit (Quit: Verlassend)
[16:31] * MarkN (~nathan@59.167.240.178) has joined #ceph
[16:47] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[16:49] * MarkN (~nathan@59.167.240.178) has joined #ceph
[17:09] <darkfade1> jantje: you'll need a separate spindle for the osd journal to get that
[17:09] <darkfade1> or one ssd for all osd journals
[17:10] <darkfade1> err, wait, was it you putting it in ram?
[17:12] <sagewk> jantje: you don't need to adjust the crush map to turn off replication, just do 'ceph osd pool set data size 1'
[17:12] <darkfade1> ahh
[17:28] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) Quit (Ping timeout: 480 seconds)
[18:01] * ghaskins_mobile (~ghaskins_@66-189-114-103.dhcp.oxfr.ma.charter.com) has joined #ceph
[18:22] <jantje> sagewk: oh, cool, thanks
[18:22] <jantje> darkfade1: maybe! And appearently I couldnt put the journal in a ramfs
[18:34] <jantje> (I might have used the wrong syntax)
[18:49] <jantje> mount failed to open journal /dev/shm/journal: Inappropriate ioctl for device
[18:49] <jantje> maybe journal is not pointing to a block device and its size wasn't configured?
[18:49] <jantje> (And yes, i configured the size)
[18:50] <gregaf> hmmm, what's your config look like?
[18:50] <jantje> ;osd journal = /dev/shm/journal
[18:50] <jantje> ;osd journal size = 500
[18:51] <jantje> under [osd]
[18:53] <jantje> I think it needs user_xattr
[18:53] <gregaf> you might need to declare the size for the OSDs individually
[18:53] <gregaf> oh, and yes, it needs user_xattr
[18:53] <jantje> tmpfs doesn't have it I think
[18:53] <gregaf> :/
[18:53] <gregaf> sorry
[18:55] * jantje off
[18:55] <jantje> cya
[18:55] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[18:57] <jantje> sagewk: it would be nice to have the journal on an tmpfs, for testing.
[18:57] <jantje> </feature-request>
[18:59] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) Quit (Quit: Osso)
[19:35] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) has joined #ceph
[19:37] * Osso_ (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) has joined #ceph
[19:37] * Osso (osso@AMontsouris-755-1-10-232.w90-46.abo.wanadoo.fr) Quit (Read error: Connection reset by peer)
[19:37] * Osso_ is now known as Osso
[19:40] <wido> could i do a little feature request for librados? A version number somewhere could be usefull when writing applications against it
[19:41] * MarkN (~nathan@59.167.240.178) has joined #ceph
[19:45] <gregaf> wido: you can put requests in the tracker, it's for features too :)
[19:45] <gregaf> I actually thought we had a version in there but now I can't find it
[19:46] <wido> gregaf: yeah, i'll open a issue for it :) but wanted to poll your opinion first
[19:46] <wido> but your anwsers says enough, i'll open one
[19:59] <wido> there was an issue in the tracker about a new crushmap destroying the performance, i can't find it, where did it go?
[19:59] <sagewk> i don't remember that one
[20:00] <gregaf> http://tracker.newdream.net/issues/282
[20:00] <gregaf> do you mean that one?
[20:01] <wido> no, not that one. It was "new crushmap destroys performance" "something isn't working as designed"
[20:03] <wido> well, what i'm seeing, i did a new mkcephfs this morning and started rsync'ing kernel.org again, which went wrong. Some OSD's started to run OOM, ok, that's clear, but ever since that happend, the cluster can't recover anymore
[20:03] <wido> OSD become so slow (that is due to the fact that they start swapping too)
[20:04] <wido> and i'm seeing #326 and #327 again
[20:06] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[20:06] <wido> It's just a shortage of RAM i guess?
[20:08] <gregaf> how much memory do your OSDs have again?
[20:09] <wido> not that much, 768 ~ 1024 per OSD max
[20:09] <wido> some machines have 4G, but run 4 OSD's
[20:09] <gregaf> but with swap on?
[20:09] <wido> yes, most of the times a few G, 6 to 8
[20:10] <wido> "load average: 265.12, 266.68, 266.59", a cosd process started to block
[20:10] <gregaf> hmmm, you're probably hitting a leak of some kind then
[20:11] <wido> probably, i'm not sure, but i saw the OOM killer a few times
[20:12] <wido> might switch to a btrfs stripe, will save some memory, might make it work better
[20:15] <gregaf> I found a number of leaks of varying sizes which I haven't merged in yet since I got overzealous with at least one of them — I'll start on figuring that out and enabling tcmalloc today
[20:16] <wido> no rush, i know i'm lacking RAM
[20:16] <wido> ofcourse, a leak could eat more RAM if i had it, but 1.5GB for two OSD's is a bit low
[20:18] <gregaf> yeah, but it shouldn't be eating so much that your OOM killer activates when you're running swap too
[20:20] <wido> true, but i'm pointing to the slow performance when recovering from a degraded cluster, that really could be due to the slow OSD's
[20:20] <wido> the OSD's with their journal on SSD really seem to outperform everything
[20:21] <gregaf> well, they would! ;)
[20:29] * MarkN (~nathan@59.167.240.178) has joined #ceph
[20:29] <wido> btw, for the RBD block device, is there a way to fully disable the kernel page cache? Why? I'm trying to setup a multipath iSCSI env on top of RBD
[20:30] <wido> you could then make a distributed block device for legacy applications (Like Windows), they could benefit from Ceph, but still use iSCSI
[20:39] <yehudasa> wido: I'm not sure why you actually want to disable the page cache
[20:40] <sagewk> btw, what iscsi target are you using? something like http://www.mail-archive.com/hail-devel@vger.kernel.org/msg00697.html?
[20:41] <wido> yehudasa: since my clients (initiators) will round robin over the iSCSI targets
[20:42] <wido> and the targets do not share a cache
[20:42] <wido> sagewk: i'm using IET: http://iscsitarget.sourceforge.net/
[20:57] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[21:26] * MarkN (~nathan@59.167.240.178) has joined #ceph
[21:44] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[21:55] * MarkN (~nathan@59.167.240.178) has joined #ceph
[22:04] * fzylogic (~fzylogic@dsl081-243-128.sfo1.dsl.speakeasy.net) has joined #ceph
[22:59] * Guest587 (quasselcor@bas11-montreal02-1128531598.dsl.bell.ca) Quit (Remote host closed the connection)
[23:01] * bbigras (quasselcor@bas11-montreal02-1128531598.dsl.bell.ca) has joined #ceph
[23:01] * bbigras is now known as Guest993
[23:13] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[23:26] * MarkN (~nathan@59.167.240.178) has joined #ceph
[23:31] * allsystemsarego (~allsystem@188.26.32.97) Quit (Quit: Leaving)
[23:34] * fzylogic (~fzylogic@dsl081-243-128.sfo1.dsl.speakeasy.net) Quit (Quit: DreamHost Web Hosting http://www.dreamhost.com)
[23:58] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.