#ceph IRC Log

Index

IRC Log for 2010-10-14

Timestamps are in GMT/BST.

[1:17] * johnl (~johnl@cpc3-brad19-2-0-cust563.barn.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[3:01] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[4:02] * greglap (~Adium@166.205.136.52) has joined #ceph
[5:04] * greglap (~Adium@166.205.136.52) Quit (Read error: Connection reset by peer)
[5:12] * greglap (~Adium@cpe-76-90-74-194.socal.res.rr.com) has joined #ceph
[5:55] * `8mycat5` (~mice8myca@66.87.0.144) has joined #ceph
[6:35] * MarkN (~nathan@59.167.240.178) Quit (Ping timeout: 480 seconds)
[6:51] * MarkN (~nathan@59.167.240.178) has joined #ceph
[7:22] * `8mycat5` (~mice8myca@66.87.0.144) Quit (Quit: xchat broken)
[7:58] * `8mycat5` (~mice8myca@66.87.2.19) has joined #ceph
[8:05] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[8:31] * hijacker (~hijacker@213.91.163.5) Quit (Remote host closed the connection)
[8:47] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[8:57] * gregorg (~Greg@78.155.152.6) Quit (Quit: Quitte)
[8:57] * gregorg (~Greg@78.155.152.6) has joined #ceph
[9:02] * hijacker (~hijacker@213.91.163.5) has joined #ceph
[9:21] * `8mycat5` (~mice8myca@66.87.2.19) Quit (Ping timeout: 480 seconds)
[9:32] * Yoric (~David@213.144.210.93) has joined #ceph
[10:00] * sentinel_e86 (~sentinel_@188.226.51.71) Quit (Quit: sh** happened)
[10:01] * sentinel_e86 (~sentinel_@188.226.51.71) has joined #ceph
[10:46] * allsystemsarego (~allsystem@188.27.167.113) has joined #ceph
[11:07] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (Quit: leaving)
[11:07] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) has joined #ceph
[11:11] * johnl (~johnl@cpc3-brad19-2-0-cust563.barn.cable.virginmedia.com) has joined #ceph
[11:54] * Jiaju (~jjzhang@222.126.194.154) Quit (Ping timeout: 480 seconds)
[11:58] * lidongyang (~lidongyan@222.126.194.154) Quit (Remote host closed the connection)
[12:00] * lidongyang (~lidongyan@222.126.194.154) has joined #ceph
[12:00] * Jiaju (~jjzhang@222.126.194.154) has joined #ceph
[13:00] * Yoric_ (~David@213.144.210.93) has joined #ceph
[13:00] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[13:00] * Yoric_ is now known as Yoric
[13:04] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[13:17] * allsystemsarego (~allsystem@188.27.167.113) Quit (Quit: Leaving)
[13:18] * allsystemsarego (~allsystem@188.27.167.113) has joined #ceph
[13:39] * Yoric (~David@213.144.210.93) has joined #ceph
[13:43] * johnl (~johnl@cpc3-brad19-2-0-cust563.barn.cable.virginmedia.com) Quit (Quit: bye)
[13:57] * Yoric_ (~David@213.144.210.93) has joined #ceph
[13:57] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[13:57] * Yoric_ is now known as Yoric
[14:00] * Yoric_ (~David@213.144.210.93) has joined #ceph
[14:00] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[14:00] * Yoric_ is now known as Yoric
[14:02] * Yoric_ (~David@213.144.210.93) has joined #ceph
[14:02] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[14:02] * Yoric_ is now known as Yoric
[14:10] * Yoric_ (~David@213.144.210.93) has joined #ceph
[14:10] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[14:10] * Yoric_ is now known as Yoric
[14:12] * Yoric_ (~David@213.144.210.93) has joined #ceph
[14:12] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[14:12] * Yoric_ is now known as Yoric
[14:14] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[14:37] * Yoric (~David@213.144.210.93) has joined #ceph
[14:41] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[14:54] * greglap (~Adium@cpe-76-90-74-194.socal.res.rr.com) Quit (Quit: Leaving.)
[14:54] * Yoric (~David@213.144.210.93) has joined #ceph
[14:58] * Yoric (~David@213.144.210.93) Quit (Read error: Connection reset by peer)
[15:09] * greglap (~Adium@166.205.137.183) has joined #ceph
[15:48] * greglap (~Adium@166.205.137.183) Quit (Read error: Connection reset by peer)
[16:18] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:05] * f4m8 is now known as f4m8_
[18:49] <wido> yehudasa: you there?
[18:50] <wido> Well, maybe later, i'll "dump" it here anyway
[18:51] <wido> I think there is some sort of memory leak in rbd or librados. Today I created a virtual machine with 1 CPU and 4GB Ram and 6 disks of 64GB. Inside the VM I used LVM to combine these disks again to one logical volume of 350GB
[18:51] <wido> on that volume I tried to sync the debian-cd archive, but while doing so, my virtual machine host, with 8GB of RAM kept going OOM, resulting in killing the VM
[18:52] <wido> beside the virtual machine (4GB) there is nothing else running on the machine
[18:52] <sagewk> wido: are you using the latest patch from the ML or git?
[18:52] <wido> inside the VM I started seeing messages like softlockup
[18:52] <wido> sagewk: latest git from about 12 hours ago
[18:52] <wido> but I saw the same about 2 days ago
[18:54] <wido> brb
[18:55] <wido> I'm not completely sure that librados or RBD is to blame, but I can't think of something else. Any ideas how to debug this?
[19:13] <yehudasa> wido: here now
[19:13] <wido> ok
[19:15] <yehudasa> wido: are you talking about the latest qemu-kvm?
[19:15] <yehudasa> I'm just debugging it
[19:16] <wido> Yes
[19:16] <wido> well, I don't want to sent you on a ghost hunt
[19:16] <wido> but that's what I'm seeing and it's pretty strange
[19:16] <wido> using qemu-kvm-0.12.3, backported the patch, but nothing strange is done
[19:17] <yehudasa> you mean it only happens with latest git version?
[19:17] <wido> well, no, I had the same thing about 2 days ago
[19:17] <wido> but this is the first time I started to do so many I/O inside a VM
[19:18] <yehudasa> yeah, well.. the one from two days ago is broken
[19:18] <yehudasa> didn't handle EAGAIN on the internal pipes
[19:19] <yehudasa> can you run the same test on the version before the recent changes?
[19:21] <yehudasa> actually, there's a quick fix for latest version
[19:21] <wido> http://ceph.newdream.net/git/?p=qemu-kvm.git;a=commit;h=2e8e9928c600ab75a82947bc35a6d8b509db78d5
[19:21] <wido> that one for example? Or older?
[19:22] <yehudasa> ah, actually, can you test with the latest that I just pushed?
[19:22] <yehudasa> that what happens when doing stuff in 12 am
[19:23] <wido> sure, you mean "rbd: fix write in aiocb" ?
[19:23] <yehudasa> yep
[19:25] <wido> compile is running right now
[20:01] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[20:17] * tjikkun (~tjikkun@195-240-122-237.ip.telfort.nl) Quit (Ping timeout: 480 seconds)
[20:18] * greglap1 (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[20:18] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Read error: Connection reset by peer)
[20:19] <wido> yehudasa: went wrong again. http://www.pastebin.org/176846
[20:23] <yehudasa> wido: looks like some memory leak
[20:23] <yehudasa> can you verify that memory footprint grows while running?
[20:25] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[20:26] <wido> not really. The process stays at 60% memory usage, and then it grows rapidly
[20:26] <wido> i'll try again
[20:29] <wido> i'm running the test again and dumping "ps aux" every 5 secs, to see if it grows that fast
[20:45] <wido> yehudasa: the memory seems to be growing slowly now, it's ate 64.5% right now, that is about 5000MB, where the VM was only allocated 4096MB
[20:45] <wido> 66.1% at the moment and growing
[20:45] <wido> 67.1%
[20:52] <yehudasa> oh, good
[20:53] <yehudasa> probably some leak
[20:54] <wido> yes, it grows to about 70% and then it gets killed
[20:54] <wido> i'll open a issue for it
[21:12] <wido> yehudasa: http://tracker.newdream.net/issues/489
[21:13] <yehudasa> great
[21:35] <wido> i'm going afk, gave you some work again with these issues ;)
[21:36] <wido> tnx and ttyl!
[21:43] * cmccabe2 (~cmccabe@dsl081-243-128.sfo1.dsl.speakeasy.net) has joined #ceph
[22:00] * allsystemsarego (~allsystem@188.27.167.113) Quit (Quit: Leaving)
[23:36] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[23:36] * greglap1 (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Read error: Connection reset by peer)
[23:52] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.