#ceph IRC Log

Index

IRC Log for 2014-07-10

Timestamps are in GMT/BST.

[0:00] * bmoloney (~oftc-webi@137.53.91.160) has joined #ceph
[0:00] <iggy> and most of the SoC's have a few pcie lanes, but hey are often used for gige or other things
[0:00] * sjm (~sjm@pool-72-76-115-220.nwrknj.fios.verizon.net) has left #ceph
[0:01] <iggy> (since most SoCs only have 100M eth built in)
[0:01] <Midnightmyth> okay
[0:01] <Serbitar> i found a soc with on chip sata/6g (1 port) and 10g e
[0:01] <Serbitar> that was about a month ago, cant remember who made it
[0:01] <iggy> oh, the new intel SoCs all have sata3
[0:02] <tnt_> https://developer.nvidia.com/jetson-tk1
[0:02] <iggy> I think 2 or 4 ports
[0:02] <tnt_> This has sata/pcie/usb3/...
[0:02] <iggy> the jetson is ridiculously expensive
[0:02] <iggy> does it have to be arm?
[0:03] <tnt_> expensive ??? I find it pretty cheap for what it has on-board.
[0:03] <iggy> I've done a lot of monetary research in this space and generally speaking, intel boards are still cheaper for >1 sata
[0:04] <bmoloney> Hello, I am looking to put together a smallish high-density ceph cluster. What are the downsides to running the MON daemons on my OSD nodes (provided there is enough CPU/RAM and some dedicated storage)?
[0:04] <Midnightmyth> I was going for inexpensive, but I guess the seagate kinetic or something similiar comes out soon
[0:04] <iggy> it is, just depends what you are after
[0:05] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) has joined #ceph
[0:05] <Midnightmyth> The jetson board is extremely expensive and most of the features useless in terms of ceph.
[0:06] <iggy> bmoloney: a lot of people do so
[0:06] <iggy> just don't go crazy and put a mon on each osd node
[0:08] <iggy> the jetson board would be more useful (ceph wise) if the eMMC was bigger (i.e. faster) and the board was cheaper
[0:08] <bmoloney> iggy: Good to hear. How many MON daemons would be recommended for 8 OSD nodes on two racks?
[0:09] <iggy> personally, I'd do 3, not sure what the "official" recommendations are
[0:09] * Cube (~Cube@12.248.40.138) Quit (Read error: Operation timed out)
[0:10] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) Quit (Read error: Operation timed out)
[0:11] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) Quit (Quit: valeech)
[0:14] <Midnightmyth> bmoloney: I know there is some info about it in the ceph docs
[0:15] <bmoloney> Midnightmyth: I saw some recommendation to separate MON and OSD daemons if possible, but not much explanation why or how big a deal it is.
[0:17] <iggy> well... if you have a single gigE connection, the OSD can saturate that
[0:17] <iggy> which could cause the MONs to reply slow and make the cluster think they've died
[0:19] <bmoloney> Ok, if anyone wants to provide feedback on my initial OSD/MON node specs I would appreciate it. Each 2U server will have: 12 cores / 24 threads @ 2.4GHz, 128GB of RAM, LSI 2208 SAS HBA, 12 6TB SAS drives for OSD (connected to SAS), 2 200GB Seagate 600 Pro SSD for journals (connected to 6Gb SATA). The network will be 10Gb Ethernet between racks and Infiniband (IP over IB) on rack.
[0:19] * markbby (~Adium@168.94.245.1) Quit (Quit: Leaving.)
[0:20] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[0:20] <Serbitar> are you using each of those 6TB disk as a separate OSD?
[0:20] <tnt_> bmoloney: how many nodes total ?
[0:20] <bmoloney> 8 nodes total
[0:21] * JayJ (~jayj@157.130.21.226) Quit (Remote host closed the connection)
[0:21] * markbby (~Adium@168.94.245.1) has joined #ceph
[0:21] <bmoloney> I was initially toying with the idea of doing RAID6 or something on the disks to reduce the number of OSD daemons, but now I am leaning towards doing one daemon per disk
[0:22] <Serbitar> k, so your journal SSDs will be a raid1?
[0:22] <tnt_> bmoloney: yeah, RAID doesn't seem like a good idea.
[0:22] <bmoloney> Serbitrar: no I was going to journal six disks on each OSD
[0:23] <tnt_> yes, 1:6 SDD vs OSD ratio is still within the recommended limits.
[0:23] <Serbitar> should be ok as long as ceph knows you have a 6 disk wide failure domain
[0:23] <tnt_> Serbitar: well, failure domain is usually the whole host.
[0:23] <Serbitar> true
[0:24] <tnt_> With 8 nodes, one SSD failure would mean a 6.25% failure which should be fine.
[0:25] <Serbitar> bmoloney: OOI what is your cost per node for that setup
[0:26] <Serbitar> i assume your racks are separated by some distance, hence the 10gE not IB between them
[0:26] <bmoloney> Serbitar: roughly $10K, it is supermicro gear
[0:26] <Serbitar> k
[0:26] <Serbitar> so thats proably USD100k for the total 576TB of disk in servers+network
[0:27] <bmoloney> yes the racks are physically seperate and my options for networking them are limited. We are basically paying to upgrade part of the campus network.
[0:27] * markbby (~Adium@168.94.245.1) Quit (Quit: Leaving.)
[0:28] <Serbitar> so you will have 2 mons in one room and 1 in the other
[0:29] <Serbitar> might be worth sorting a spare mon on each side which is disabled
[0:29] <tnt_> bmoloney: can you bundle a couple of 10G link together ?
[0:29] <Serbitar> in case a room burns down
[0:30] <Midnightmyth> I wonder how cheap you can build a ceph setup with arm boards
[0:30] <Midnightmyth> I dont need performance for my setup at all..
[0:30] <tnt_> or put a 3rd mon completely elsewhere. mon traffic is minimal.
[0:30] * bandrus (~Adium@79.102.121.173) Quit (Quit: Leaving.)
[0:30] <Serbitar> tnt_: true, could be in a separate room on just 1g ethernet fibre or otherwise
[0:30] <bmoloney> Serbitrar: Would I have to manually start up these mons in this case? Or is there some built in functionality to have some inactive mon daemons waiting for a failure?
[0:30] <Serbitar> i would keep it manually
[0:31] <bmoloney> tnt_: Possibly, but we are already paying quite a bit since the IT here will only use Cisco gear
[0:31] <Serbitar> because the cluster cant know for sure wether someone has unplugged the network by accident or the room is actually irrecoverable
[0:33] <Serbitar> but having a node in a separate room sounds like a better idea
[0:33] <Serbitar> mon node
[0:33] * reed (~reed@75-101-54-131.dsl.static.sonic.net) Quit (Quit: Ex-Chat)
[0:33] <bmoloney> Serbitar: So I guess if the rack with 2 mons is lost, the cluster will lose quorum. Unless I have a seperate mon running elsewhere
[0:33] <Serbitar> tnt_: you could run that mon on a small atom box right?
[0:34] <Serbitar> well, with 2 network ports
[0:34] * rendar (~I@host208-176-dynamic.23-79-r.retail.telecomitalia.it) Quit ()
[0:35] <bmoloney> So run one mon on each rack, and one somewhere else on the network with a 1Gb ethernet link
[0:35] <tnt_> Serbitar: I'ts pretty light in CPU. But it actually does continuous IO and can take a bit of memory when you're doing recovery on tens of thousands of PGs.
[0:36] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[0:36] * rturk|afk is now known as rturk
[0:36] <tnt_> and being even a bit too slow on theses IO will cause it to be ejected from the quorum from time to time.
[0:36] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[0:37] <Serbitar> k
[0:38] * AfC (~andrew@nat-gw2.syd4.anchor.net.au) has joined #ceph
[0:39] <bmoloney> tnt_: How sensitive is this to latency? We could pay to put a mon in a nearby datacenter, which should have close to 10G Ethernet speeds but worse latency than between our two racks.
[0:44] <tnt_> bmoloney: I don't think that 10's of ms would make much of a difference. But better check with mon experts.
[0:44] <tnt_> joao: ^^
[0:46] <bmoloney> Also, I am still not 100% sure where to put the OS data. I guess this shouldn't be accessed that much after boot? I was thinking maybe carving a small partion out of each OSD disk and doing RAID10 or something.
[0:46] * Cube (~Cube@66-87-67-96.pools.spcsdns.net) has joined #ceph
[0:48] <tnt_> bmoloney: that's what I did, but not sure it's recommended. I pretty much had the same reasons as you said ...
[0:49] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[0:50] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[0:50] <bmoloney> tnt_: Are you running your mon daemons on your OSD nodes as well? Do the mon daemons need more disk i/o?
[0:52] <tnt_> Yes, they need disk IO. I have them on separate disk. Not much bandwidth, but enough IOPS to ruin your OSD if you put it on the same disks.
[0:52] * zack_dolby (~textual@p67f6b6.tokynt01.ap.so-net.ne.jp) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[0:54] <bmoloney> I will need to think about it some more then, I do still have open PCIe slots to work with...
[0:56] <joao> 10ms is not that bad for the monitors
[0:58] <joao> if you get to 25 ms latency you may start seeing some warnings though
[0:58] <joao> but that's adjustable if you feel like your cluster is dealing fine with your latency
[0:59] * kevinc (~kevinc__@client65-40.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[1:00] <bmoloney> @joao: Thanks!
[1:02] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[1:03] <Serbitar> bmoloney: can you go with some cheap 1u boxes with 2x1ge and say an i3? single small SSD
[1:03] <iggy> I'd say split the SSDs up and then use one slice of each (mirrored) for the the mon data
[1:04] <Serbitar> for the mons
[1:05] * KaZeR (~kazer@64.201.252.132) Quit (Ping timeout: 480 seconds)
[1:06] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) has joined #ceph
[1:06] <bmoloney> Serbitrar: yes I am considering it, but even a cheap 1U box is going to be around $1K
[1:06] <Serbitar> yeah
[1:06] * hasues (~hasues@kwfw01.scrippsnetworksinteractive.com) Quit (Quit: Leaving.)
[1:06] <Serbitar> OS on usb stick for the OSD nodes?
[1:07] <Serbitar> a lto of servers have internal usb ports
[1:08] <bmoloney> iggy: The journal SSD bandwidth is already a bit under the combined bandwidth of 6 OSD drives. Those seagate 6TB drives have some pretty impresive sequential BW numbers.
[1:08] <Serbitar> i have also seen pci-e cards which have a 2.5"hdd hotplug out the back
[1:09] * rweeks (~rweeks@pat.hitachigst.com) Quit (Quit: Leaving)
[1:09] <bmoloney> Serbitrar: yes, or even maybe a cheap PCIe SSD just for OS and mon data.
[1:10] * KaZeR (~kazer@64.201.252.132) has joined #ceph
[1:14] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:14] <bmoloney> Or do the reverse and use a PCIe SSD for journals and the 2.5" hotswap bays for OS/mon data. Does anyone know a PCIe SSD with enough endurance that won't break the bank? Those 200GB 600 pro drives have 540TB of write endurance and only cost ~$300
[1:15] * Nats_ (~Nats@2001:8000:200c:0:d525:8330:e93b:70f2) has joined #ceph
[1:15] * huangjun (~kvirc@117.151.41.96) Quit (Read error: Operation timed out)
[1:18] * kevinc (~kevinc__@client65-40.sdsc.edu) has joined #ceph
[1:18] * CAPSLOCK2000 (~oftc@2001:610:748:1::8) Quit (Ping timeout: 480 seconds)
[1:22] * Nats__ (~Nats@2001:8000:200c:0:ed20:896c:d0a4:9f54) Quit (Ping timeout: 480 seconds)
[1:41] * oms101 (~oms101@p20030057EA050C00EEF4BBFFFE0F7062.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:45] * danieagle (~Daniel@179.184.165.184.static.gvt.net.br) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[1:46] <iggy> how much sequential i/o do you actually do?
[1:46] * rturk is now known as rturk|afk
[1:47] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) has joined #ceph
[1:50] * oms101 (~oms101@p20030057EA048100EEF4BBFFFE0F7062.dip0.t-ipconnect.de) has joined #ceph
[1:53] <bmoloney> iggy: Our workload is all over the map. I am basically trying to provide a large pool of HA storage that a bunch of different users/groups will then use as they need. I don't need tons of IOPs for a massive database or anything like that though.
[1:56] * huangjun (~kvirc@58.49.113.200) has joined #ceph
[1:58] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) has joined #ceph
[1:59] <iggy> I was just making the point that another user of the OSDs might not make a huge difference
[2:05] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[2:07] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) has joined #ceph
[2:11] * adamcrume (~quassel@2601:9:6680:47:30c0:25bc:69f7:d10f) Quit (Remote host closed the connection)
[2:13] * LeaChim (~LeaChim@host86-161-90-156.range86-161.btcentralplus.com) Quit (Read error: Operation timed out)
[2:13] * rturk|afk is now known as rturk
[2:13] * sarob (~sarob@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[2:14] * rturk is now known as rturk|afk
[2:15] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[2:21] * kevinc (~kevinc__@client65-40.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[2:27] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) Quit (Quit: Leaving.)
[2:28] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) Quit (Ping timeout: 480 seconds)
[2:29] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) has joined #ceph
[2:30] * xarses (~andreww@12.164.168.117) Quit (Ping timeout: 480 seconds)
[2:30] * bmoloney (~oftc-webi@137.53.91.160) Quit (Quit: Page closed)
[2:43] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Quit: Leaving.)
[2:48] * Cube1 (~Cube@66-87-131-123.pools.spcsdns.net) has joined #ceph
[2:53] * Cube (~Cube@66-87-67-96.pools.spcsdns.net) Quit (Ping timeout: 480 seconds)
[2:59] * KaZeR (~kazer@64.201.252.132) Quit (Remote host closed the connection)
[3:00] * xarses (~andreww@c-76-126-112-92.hsd1.ca.comcast.net) has joined #ceph
[3:01] * lofejndif (~lsqavnbok@7DKAAA80M.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[3:04] * lucas1 (~Thunderbi@222.247.57.50) has joined #ceph
[3:07] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) has joined #ceph
[3:09] * narb (~Jeff@38.99.52.10) Quit (Quit: narb)
[3:15] * JC (~JC@AMontpellier-651-1-32-204.w90-57.abo.wanadoo.fr) Quit (Quit: Leaving.)
[3:15] * KaZeR (~kazer@c-67-161-64-186.hsd1.ca.comcast.net) has joined #ceph
[3:16] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[3:20] * capri (~capri@212.218.127.222) Quit (Read error: Connection reset by peer)
[3:29] * zerick (~eocrospom@190.187.21.53) Quit (Read error: Operation timed out)
[3:35] * sigsegv (~sigsegv@188.25.121.7) has left #ceph
[3:43] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[3:46] * zhaochao (~zhaochao@123.151.134.238) has joined #ceph
[3:49] * stein (~stein@91.247.228.48) Quit (Remote host closed the connection)
[3:54] * stein (~stein@91.247.228.48) has joined #ceph
[3:55] * al (d@niel.cx) Quit (Ping timeout: 480 seconds)
[3:59] * al (d@niel.cx) has joined #ceph
[4:00] * lucas1 (~Thunderbi@222.247.57.50) Quit (Quit: lucas1)
[4:04] * huangjun (~kvirc@58.49.113.200) Quit (Ping timeout: 480 seconds)
[4:09] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) has joined #ceph
[4:15] * shang (~ShangWu@175.41.48.77) has joined #ceph
[4:19] * angdraug (~angdraug@12.164.168.117) Quit (Quit: Leaving)
[4:20] * KevinPerks (~Adium@cpe-174-098-096-200.triad.res.rr.com) Quit (Quit: Leaving.)
[4:23] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) Quit (Ping timeout: 480 seconds)
[4:23] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) has joined #ceph
[4:35] * bkopilov (~bkopilov@213.57.16.133) Quit (Read error: Operation timed out)
[4:35] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) Quit (Read error: Connection reset by peer)
[4:35] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) has joined #ceph
[4:38] * ghost1 (~pablodelg@c-174-61-25-255.hsd1.fl.comcast.net) Quit (Quit: ghost1)
[4:45] * sjm (~sjm@pool-72-76-115-220.nwrknj.fios.verizon.net) has joined #ceph
[4:46] * haomaiwang (~haomaiwan@118.186.129.94) Quit (Remote host closed the connection)
[4:46] * haomaiwang (~haomaiwan@124.248.205.19) has joined #ceph
[4:56] * diegows (~diegows@190.190.5.238) Quit (Ping timeout: 480 seconds)
[5:01] * haomaiwa_ (~haomaiwan@118.186.129.94) has joined #ceph
[5:06] * haomaiwang (~haomaiwan@124.248.205.19) Quit (Read error: Operation timed out)
[5:09] * KaZeR (~kazer@c-67-161-64-186.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[5:10] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) has joined #ceph
[5:11] * ccourtaut_ (~ccourtaut@ks362468.kimsufi.com) Quit (Remote host closed the connection)
[5:16] * yguang11 (~yguang11@2406:2000:ef96:e:f556:a6e2:5ace:a693) Quit (Ping timeout: 480 seconds)
[5:18] * madkiss (~madkiss@p5795B342.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[5:20] * ccourtaut_ (~ccourtaut@2001:41d0:1:eed3::1) has joined #ceph
[5:31] * yguang11 (~yguang11@2406:2000:ef96:e:15:149c:8a03:65d5) has joined #ceph
[5:38] * Psi-Jack_ (~Psi-Jack@psi-jack.user.oftc.net) has joined #ceph
[5:39] * Psi-Jack_ (~Psi-Jack@psi-jack.user.oftc.net) Quit ()
[5:42] * aarcane (~aarcane@99-42-64-118.lightspeed.irvnca.sbcglobal.net) Quit (Quit: Leaving)
[5:53] * Vacum_ (~vovo@88.130.222.18) has joined #ceph
[5:53] * vbellur1 (~vijay@122.167.82.113) has joined #ceph
[5:59] * vbellur (~vijay@122.167.227.36) Quit (Ping timeout: 480 seconds)
[6:00] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[6:00] * Vacum (~vovo@88.130.217.235) Quit (Ping timeout: 480 seconds)
[6:01] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[6:01] * yguang11 (~yguang11@2406:2000:ef96:e:15:149c:8a03:65d5) Quit (Remote host closed the connection)
[6:02] * yguang11 (~yguang11@2406:2000:ef96:e:15:149c:8a03:65d5) has joined #ceph
[6:08] * AfC (~andrew@nat-gw2.syd4.anchor.net.au) Quit (Ping timeout: 480 seconds)
[6:11] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) has joined #ceph
[6:16] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:16] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[6:16] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) has joined #ceph
[6:19] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[6:26] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) Quit (Quit: valeech)
[6:28] * theanalyst (~abhi@49.32.0.29) has joined #ceph
[6:37] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) Quit (Quit: Leaving.)
[6:48] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[6:53] * mjeanson (~mjeanson@00012705.user.oftc.net) Quit (Remote host closed the connection)
[6:54] * mjeanson (~mjeanson@bell.multivax.ca) has joined #ceph
[7:00] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) Quit (Ping timeout: 480 seconds)
[7:09] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[7:09] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) has joined #ceph
[7:09] * ChanServ sets mode +o elder
[7:11] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) has joined #ceph
[7:19] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[7:22] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) Quit (Quit: Leaving.)
[7:26] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) has joined #ceph
[7:28] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) Quit (Read error: Connection reset by peer)
[7:34] * b0e (~aledermue@juniper1.netways.de) has joined #ceph
[7:34] * rdas (~rdas@121.244.87.115) has joined #ceph
[7:39] * vbellur1 (~vijay@122.167.82.113) Quit (Ping timeout: 480 seconds)
[7:44] * AfC (~andrew@nat-gw2.syd4.anchor.net.au) has joined #ceph
[7:51] * hasues (~hazuez@108-236-232-243.lightspeed.knvltn.sbcglobal.net) Quit (Quit: Leaving.)
[7:54] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[7:55] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit ()
[7:56] * vbellur (~vijay@121.244.87.117) has joined #ceph
[7:58] * saurabh (~saurabh@121.244.87.117) has joined #ceph
[8:00] * drankis (~drankis__@159.148.207.145) has joined #ceph
[8:08] * drankis (~drankis__@159.148.207.145) Quit (Ping timeout: 480 seconds)
[8:11] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) has joined #ceph
[8:13] * ikrstic (~ikrstic@178-221-67-116.dynamic.isp.telekom.rs) has joined #ceph
[8:19] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[8:26] * drankis (~drankis__@89.111.13.198) has joined #ceph
[8:34] * lalatenduM (~lalatendu@121.244.87.117) has joined #ceph
[8:40] * haomaiwa_ (~haomaiwan@118.186.129.94) Quit (Remote host closed the connection)
[8:41] * haomaiwang (~haomaiwan@124.248.205.19) has joined #ceph
[8:42] * rendar (~I@95.238.179.240) has joined #ceph
[8:43] * ikrstic (~ikrstic@178-221-67-116.dynamic.isp.telekom.rs) Quit (Read error: Operation timed out)
[8:43] * ikrstic_ (~ikrstic@109-93-26-12.dynamic.isp.telekom.rs) has joined #ceph
[8:48] * Sysadmin88 (~IceChat77@94.4.20.0) Quit (Quit: On the other hand, you have different fingers.)
[8:53] * yguang11 (~yguang11@2406:2000:ef96:e:15:149c:8a03:65d5) Quit (Remote host closed the connection)
[8:54] * yguang11 (~yguang11@vpn-nat.peking.corp.yahoo.com) has joined #ceph
[8:54] * yguang11 (~yguang11@vpn-nat.peking.corp.yahoo.com) Quit (Read error: Connection reset by peer)
[8:55] * yguang11 (~yguang11@2406:2000:ef96:e:29b0:eeef:abbf:70fb) has joined #ceph
[8:57] * ScOut3R (~ScOut3R@catv-89-133-22-210.catv.broadband.hu) has joined #ceph
[9:02] * madkiss (~madkiss@ws14-141.sys11.net) has joined #ceph
[9:04] * ismell (~ismell@host-64-17-89-79.beyondbb.com) Quit (Ping timeout: 480 seconds)
[9:05] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Read error: Connection reset by peer)
[9:06] * Hell_Fire__ (~HellFire@123-243-155-184.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[9:06] * MACscr (~Adium@c-50-158-183-38.hsd1.il.comcast.net) has joined #ceph
[9:07] * AfC (~andrew@nat-gw2.syd4.anchor.net.au) Quit (Ping timeout: 480 seconds)
[9:07] * vbellur (~vijay@121.244.87.117) Quit (Ping timeout: 480 seconds)
[9:08] * Hell_Fire__ (~HellFire@123-243-155-184.static.tpgi.com.au) has joined #ceph
[9:11] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[9:12] * haomaiwa_ (~haomaiwan@118.186.129.94) has joined #ceph
[9:13] * analbeard (~shw@support.memset.com) has joined #ceph
[9:18] * haomaiwang (~haomaiwan@124.248.205.19) Quit (Read error: Operation timed out)
[9:22] * vbellur (~vijay@121.244.87.124) has joined #ceph
[9:23] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[9:23] * michalefty (~micha@188-195-129-145-dynip.superkabel.de) has joined #ceph
[9:24] * sputnik1_ (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[9:27] * dgarcia_ (~dgarcia@50-73-137-146-ip-static.hfc.comcastbusiness.net) Quit (Quit: Leaving)
[9:27] * yguang11_ (~yguang11@vpn-nat.peking.corp.yahoo.com) has joined #ceph
[9:31] * yguang11 (~yguang11@2406:2000:ef96:e:29b0:eeef:abbf:70fb) Quit (Ping timeout: 480 seconds)
[9:33] * hyperbaba (~hyperbaba@private.neobee.net) has joined #ceph
[9:36] * yguang11 (~yguang11@vpn-nat.peking.corp.yahoo.com) has joined #ceph
[9:36] * yguang11_ (~yguang11@vpn-nat.peking.corp.yahoo.com) Quit (Read error: Connection reset by peer)
[9:39] * trond (~trond@evil-server.alseth.info) has joined #ceph
[9:40] * ismell (~ismell@host-24-56-188-10.beyondbb.com) has joined #ceph
[9:50] * pressureman (~pressurem@62.217.45.26) Quit (Ping timeout: 480 seconds)
[9:51] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has joined #ceph
[9:51] * ChanServ sets mode +v andreask
[9:52] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[9:55] <nyerup> I'm seeing rapid memory consumption on OSDs during snapshot removal.
[9:56] <nyerup> I'm running 0.67.7-74-gb2ac719, built from the wip-throttle-snap patch to workaround snapshot removal causing full client I/O stalls.
[9:57] <nyerup> I take a weekly snapshot of my pool, and keep a 30 day retention of the snapshots.
[9:59] <nyerup> About an hour after I start removing a snapshot, memory starts climbing with about 3G per hour on data nodes running 12 OSDs each.
[9:59] <nyerup> It keeps climbing like that until the removal is complete.
[10:00] <nyerup> And as this takes several hours, my data nodes OOM before they reach that point, unless I restart the OSDs during the process.
[10:00] <nyerup> Only OSDs holding primary PGs see this symptom.
[10:02] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[10:02] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has joined #ceph
[10:02] * zack_dol_ (~textual@e0109-114-22-8-147.uqwimax.jp) has joined #ceph
[10:03] <nyerup> https://jespernyerup.dk/misc/osd-byte-avail-vs-osd-mem-active.png
[10:04] <nyerup> That's a plot of 40 days history of free diskspace on OSDs (blue) and active memory on data nodes (green).
[10:05] * zack_dol_ (~textual@e0109-114-22-8-147.uqwimax.jp) Quit ()
[10:05] <tnt_> 3G, is that per OSD process or for all of them ?
[10:05] <nyerup> As you see, every time free diskspace increases ??? as a result of snapshot removal ??? memory goes berserk.
[10:05] <nyerup> tnt_: That's total for 12 OSDs.
[10:06] * zack_dolby (~textual@e0109-114-22-8-147.uqwimax.jp) Quit (Ping timeout: 480 seconds)
[10:07] <tnt_> So each of your OSD process takes like 3G of ram when IDLE ?
[10:08] <nyerup> When they're not removing snapshots, yes. They still serve I/O, so they're not exactly idle.
[10:08] * shang (~ShangWu@175.41.48.77) Quit (Remote host closed the connection)
[10:09] <nyerup> https://jespernyerup.dk/misc/osd-byte-avail-vs-osd-mem-active-2days.png
[10:09] <tnt_> right but still ... 3G is a pretty big baseline.
[10:09] <nyerup> This is the same plot, just for the last 48 hours.
[10:10] * shang (~ShangWu@27.100.18.200) has joined #ceph
[10:10] <cookednoodles> nyerup, you should consider reporting it
[10:11] <tnt_> Yeah, OSD memory usage is getting a bit out of hand ...
[10:11] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has left #ceph
[10:11] <nyerup> tnt_: It's a somewhat big cluster.
[10:11] <rendar> nyerup: what is 'osd' ?
[10:11] <tnt_> nyerup: when doing a heap stats in the middle of the snapshot, do you see ltos of memory freeable ?
[10:12] <nyerup> tnt_: It's ongoing now ??? can you tell me how to check that?
[10:12] <tnt_> nyerup: well, memory should only be a function of the # of PGs per OSD. How many do you have per OSD ?
[10:12] <nyerup> 66560 pgs, on 936 osds.
[10:12] <nyerup> rendar: That's an Object Storage Daemon. :)
[10:13] <rendar> i see
[10:13] <tnt_> So that's "only" like 150 PGs per OSD (assuming a replication of 2)
[10:13] <nyerup> Repfac is 3.
[10:14] <nyerup> So ~210.
[10:14] <tnt_> Yup. Still not very big, so that memory definitely increase quite a lot with the # of objects stored as well which I thought it shouldn't .
[10:15] <nyerup> cookednoodles: Yeah, I just wanted to check if any of you clever dudes had some great ideas. :)
[10:15] <tnt_> nyerup: do the heap stats, and possibly heap release to see if it decreases much. This might be a better workaround that a full osd restart.
[10:15] <nyerup> tnt_: How do I check the heap stats?
[10:16] <tnt_> ceph tell osd.X heap stats IIRC.
[10:16] <tnt_> http://ceph.com/docs/master/rados/troubleshooting/memory-profiling/
[10:16] <nyerup> Cool. I'll give it a go.
[10:16] <joao> nyerup, can you please try 'ceph tell osd.X heap stats', for some osd.X that has its memory growing up?
[10:17] <joao> oh, tnt_ had already said this
[10:17] <joao> nice
[10:17] <tnt_> :)
[10:17] <joao> also, morning all
[10:18] * lalatenduM (~lalatendu@121.244.87.117) Quit (Quit: Leaving)
[10:19] <nyerup> https://jespernyerup.dk/misc/osd-heap-stats.txt
[10:19] <nyerup> tnt_, joao: ^
[10:19] <nyerup> joao: Good morning. :)
[10:20] <joao> nyerup, try the same command but s/stats/release/
[10:20] <joao> and then run stats again
[10:20] <rendar> does ceph use *always* ssh?
[10:21] <joao> I don't think ceph uses ssh at all
[10:21] * leseb (~leseb@81-64-215-19.rev.numericable.fr) Quit (Ping timeout: 480 seconds)
[10:21] <rendar> i was reading here http://www.sebastien-han.fr/blog/2012/06/10/introducing-ceph-to-openstack/ and the author says that every mkcephfs makes a ssh call
[10:22] <joao> ah
[10:22] <joao> so does ceph-deploy
[10:22] <cookednoodles> also you're reading a 2 year old post :P
[10:22] <joao> deployment tools do make ssh calls
[10:23] <rendar> cookednoodles: oh, so is that old stuff?
[10:23] <nyerup> joao: https://jespernyerup.dk/misc/osd-heap-release-stats.txt
[10:23] <joao> ceph itself doesn't use ssh at all
[10:23] <rendar> joao: i see, thanks
[10:23] <cookednoodles> https://github.com/ceph/ceph-deploy
[10:23] <cookednoodles> uses ssh
[10:23] <cookednoodles> ceph its self doesn't
[10:23] <rendar> ok
[10:24] <rendar> but, it could use ssh or ssl connections, if i configure ceph to do that also for data, right?
[10:24] <cookednoodles> o.O
[10:24] <cookednoodles> What are you trying to do ?
[10:25] <nyerup> joao: To my untrained eyes, that doesn't look like it did me any good. :)
[10:25] <rendar> cookednoodles: when i use ceph for data (and not for deploying), basically i'm using ceph which is sending my files data over the cluster, can i use ssl for that?
[10:26] <cookednoodles> well why would you ?
[10:26] <cookednoodles> good practice dictates that you'd have it on an isolated network or vlan
[10:26] <cookednoodles> the whole need for encryption is moot
[10:26] * pressureman (~pressurem@62.217.45.26) Quit (Quit: Ex-Chat)
[10:26] <rendar> cookednoodles: for example i don't trust other machines inside my network
[10:27] <cookednoodles> but thats why you have a vlan or isolated network
[10:27] <cookednoodles> and only put storage on that network
[10:27] * jtang_ (~jtang@80.111.83.231) Quit (Remote host closed the connection)
[10:27] <rendar> cookednoodles: ok
[10:28] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[10:28] * jtang_ (~jtang@80.111.83.231) has joined #ceph
[10:29] * marrusl (~mark@faun.canonical.com) has joined #ceph
[10:29] <joao> nyerup, I was hoping tcmalloc happened to be holding off on pages to be released to the osd
[10:29] <joao> that doesn't seem to be the case
[10:30] <nyerup> joao: I see.
[10:30] <joao> it would be great if you could follow the memory profiling doc
[10:30] <joao> certainly would provide more insight on what's happening
[10:32] <nyerup> joao: As in `ceph osd tell osd.X heap start_profiler`?
[10:32] <joao> yeah
[10:32] <nyerup> How long should I keep the profiler running?
[10:32] <joao> there's no good estimate
[10:32] <nyerup> I'm assuming it'll have some sort of performance impact on the OSD it's running on?
[10:33] * ikrstic_ (~ikrstic@109-93-26-12.dynamic.isp.telekom.rs) Quit (Read error: Operation timed out)
[10:33] <joao> that's a fair assumption, but I don't know how much it will be
[10:33] <joao> or whether it's negligible
[10:33] <nyerup> OK. I'll give it a go, and keep an eye out.
[10:34] <joao> if you see performance dropping just dump and stop
[10:34] <joao> if memory consumption isn't growing still it may not even catch the culprit
[10:35] <joao> also, you'll certainly want to perform 'ceph tell osd.X' instead of 'ceph osd tell X' (as seen in the docs)
[10:35] * vbellur (~vijay@121.244.87.124) Quit (Read error: Operation timed out)
[10:36] * ikrstic_ (~ikrstic@93-86-144-171.dynamic.isp.telekom.rs) has joined #ceph
[10:37] <nyerup> joao: Yes, sorry. Typo. :)
[10:37] <nyerup> I see the OSDs CPU usage increasing from ~10% of a core to ~110%.
[10:38] <nyerup> But it appears to still serve I/O.
[10:38] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Ping timeout: 480 seconds)
[10:38] * lcavassa (~lcavassa@78.25.240.221) has joined #ceph
[10:38] <joao> well, I'm going afk for a bit
[10:38] <joao> hope it works out; I'll be back soon
[10:41] <nyerup> Is the profile output a ring buffer? I see a 1M logfile, and it doesn't appear to be growing.
[10:43] * sjm (~sjm@pool-72-76-115-220.nwrknj.fios.verizon.net) Quit (Read error: Operation timed out)
[10:44] <nyerup> Oh. There came another log file.
[10:45] * marrusl (~mark@faun.canonical.com) Quit (Ping timeout: 480 seconds)
[10:48] * vbellur (~vijay@121.244.87.117) has joined #ceph
[11:01] * Nats_ (~Nats@2001:8000:200c:0:d525:8330:e93b:70f2) Quit (Read error: Connection reset by peer)
[11:02] * Nats_ (~Nats@2001:8000:200c:0:d525:8330:e93b:70f2) has joined #ceph
[11:03] * yguang11_ (~yguang11@vpn-nat.peking.corp.yahoo.com) has joined #ceph
[11:03] * yguang11 (~yguang11@vpn-nat.peking.corp.yahoo.com) Quit (Read error: Connection reset by peer)
[11:03] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[11:03] * salgeras (~salgeras@sw4i.wifi.b92.net) has joined #ceph
[11:03] * Qu310 (~Qu310@ip-121-0-1-110.static.dsl.onqcomms.net) Quit (Remote host closed the connection)
[11:04] * Qu310 (~Qu310@ip-121-0-1-110.static.dsl.onqcomms.net) has joined #ceph
[11:04] * mjeanson (~mjeanson@00012705.user.oftc.net) Quit (Remote host closed the connection)
[11:04] * zigo_ (quasselcor@ipv6-ftp.gplhost.com) has joined #ceph
[11:04] * nolan (~nolan@2001:470:1:41:a800:ff:fe3e:ad08) Quit (Remote host closed the connection)
[11:04] * nolan (~nolan@2001:470:1:41:a800:ff:fe3e:ad08) has joined #ceph
[11:05] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (Remote host closed the connection)
[11:05] * mjeanson (~mjeanson@bell.multivax.ca) has joined #ceph
[11:07] * _are_ (~quassel@h1417489.stratoserver.net) has joined #ceph
[11:08] * zigo (quasselcor@ipv6-ftp.gplhost.com) Quit (Read error: Connection reset by peer)
[11:08] * yguang11_ (~yguang11@vpn-nat.peking.corp.yahoo.com) Quit (Read error: Connection reset by peer)
[11:08] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[11:09] * yguang11 (~yguang11@2406:2000:ef96:e:983b:6900:13a:a1d4) has joined #ceph
[11:09] * marrusl (~mark@faun.canonical.com) has joined #ceph
[11:09] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[11:10] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[11:11] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[11:11] * bandrus (~Adium@h-199-142.a137.corp.bahnhof.se) has joined #ceph
[11:11] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[11:11] <nyerup> tnt_, joao: For the record, I'm trying to keep the profile running until the snapshot removal is completed. I'll return once this happens.
[11:12] <nyerup> profiler*
[11:13] <salgeras> hi guys. i'm having a problem with one of my ceph-mon nodes. occasionally "cephmon3" node drops out of the quorum. "cephmon1" (leader) node sees it like this:
[11:13] <salgeras> mon.cephmon1@0(leader).elector(26754) discarding election message: 10.0.0.51:6800/0 not in my monmap e2: 3 mons at {cephmon1=10.0.0.47:6789/0,cephmon2=10.0.0.48:6789/0,cephmon3=0.0.0.0:6800/0}
[11:14] * sm1ly (~sm1ly@broadband-77-37-240-109.nationalcablenetworks.ru) has joined #ceph
[11:14] <salgeras> any ideas why..?
[11:14] * sm1ly (~sm1ly@broadband-77-37-240-109.nationalcablenetworks.ru) Quit ()
[11:15] * jamespd (~mucky@mucky.socket7.org) Quit (Read error: Connection reset by peer)
[11:15] * jamespd (~mucky@mucky.socket7.org) has joined #ceph
[11:16] * JC (~JC@AMontpellier-651-1-32-204.w90-57.abo.wanadoo.fr) has joined #ceph
[11:16] * nljmo (~nljmo@5ED6C263.cm-7-7d.dynamic.ziggo.nl) has joined #ceph
[11:16] <joao> salgeras, http://tracker.ceph.com/issues/5804
[11:17] * shang (~ShangWu@27.100.18.200) Quit (Ping timeout: 480 seconds)
[11:19] * marrusl_ (~mark@faun.canonical.com) has joined #ceph
[11:19] <salgeras> joao: tnx for the link. will check it
[11:25] * yguang11 (~yguang11@2406:2000:ef96:e:983b:6900:13a:a1d4) Quit (Remote host closed the connection)
[11:26] * yguang11 (~yguang11@2406:2000:ef96:e:983b:6900:13a:a1d4) has joined #ceph
[11:26] * haomaiwa_ (~haomaiwan@118.186.129.94) Quit (Remote host closed the connection)
[11:27] * haomaiwang (~haomaiwan@124.248.205.19) has joined #ceph
[11:30] * shang (~ShangWu@175.41.48.77) has joined #ceph
[11:31] * analbeard (~shw@185.28.167.198) has joined #ceph
[11:33] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) has joined #ceph
[11:34] * yguang11 (~yguang11@2406:2000:ef96:e:983b:6900:13a:a1d4) Quit (Ping timeout: 480 seconds)
[11:35] * i_m (~ivan.miro@gbibp9ph1--blueice4n1.emea.ibm.com) has joined #ceph
[11:37] * bandrus (~Adium@h-199-142.a137.corp.bahnhof.se) Quit (Quit: Leaving.)
[11:40] * danieljh (~daniel@0001b4e9.user.oftc.net) Quit (Remote host closed the connection)
[11:45] * bandrus (~Adium@c-4f66f526-74736162.cust.telenor.se) has joined #ceph
[11:45] * bandrus (~Adium@c-4f66f526-74736162.cust.telenor.se) Quit ()
[11:46] * Cube (~Cube@66-87-67-122.pools.spcsdns.net) has joined #ceph
[11:46] * Cube1 (~Cube@66-87-131-123.pools.spcsdns.net) Quit (Read error: Connection reset by peer)
[11:50] * marrusl (~mark@faun.canonical.com) Quit (Remote host closed the connection)
[11:50] * marrusl_ (~mark@faun.canonical.com) Quit (Remote host closed the connection)
[11:50] * marrusl (~mark@faun.canonical.com) has joined #ceph
[11:50] * imriz (~imriz@5.29.200.177) Quit (Ping timeout: 480 seconds)
[11:51] * AfC (~andrew@2001:44b8:31cb:d400:6e88:14ff:fe33:2a9c) has joined #ceph
[11:57] * danieljh (~daniel@0001b4e9.user.oftc.net) has joined #ceph
[11:58] * hyperbaba_ (~hyperbaba@private.neobee.net) has joined #ceph
[11:59] * Cube (~Cube@66-87-67-122.pools.spcsdns.net) Quit (Quit: Leaving.)
[11:59] * hyperbaba (~hyperbaba@private.neobee.net) Quit (Read error: No route to host)
[12:02] <salgeras> joao: tnx again, it worked after recreating the cephmon3 node (using ceph-deploy). HEALTH_OK
[12:03] * hyperbaba__ (~hyperbaba@private.neobee.net) has joined #ceph
[12:06] * haomaiwa_ (~haomaiwan@118.186.129.94) has joined #ceph
[12:07] * hyperbaba_ (~hyperbaba@private.neobee.net) Quit (Ping timeout: 480 seconds)
[12:08] * ScOut3R (~ScOut3R@catv-89-133-22-210.catv.broadband.hu) Quit (Read error: Operation timed out)
[12:10] * s3an2 (~root@korn.s3an.me.uk) Quit (Read error: Operation timed out)
[12:11] * garphy`aw (~garphy@frank.zone84.net) Quit (Ping timeout: 480 seconds)
[12:12] * [fred] (fred@earthli.ng) Quit (Ping timeout: 480 seconds)
[12:12] * haomaiwang (~haomaiwan@124.248.205.19) Quit (Read error: Operation timed out)
[12:12] * DrWhax (~whx@000199fa.user.oftc.net) Quit (Remote host closed the connection)
[12:13] * madkiss (~madkiss@ws14-141.sys11.net) Quit (Quit: Leaving.)
[12:13] * al (d@niel.cx) Quit (Ping timeout: 480 seconds)
[12:13] * al (d@niel.cx) has joined #ceph
[12:13] * garphy`aw (~garphy@frank.zone84.net) has joined #ceph
[12:13] * Amto_res (~amto_res@ks312256.kimsufi.com) Quit (Read error: Connection reset by peer)
[12:13] * ssejourne (~ssejourne@37.187.216.206) Quit (Read error: Connection reset by peer)
[12:13] * raso (~raso@deb-multimedia.org) Quit (Ping timeout: 480 seconds)
[12:14] * [fred] (fred@earthli.ng) has joined #ceph
[12:14] * raso (~raso@deb-multimedia.org) has joined #ceph
[12:15] * ScOut3R (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) has joined #ceph
[12:15] * mjeanson (~mjeanson@00012705.user.oftc.net) Quit (Remote host closed the connection)
[12:16] * s3an2 (~root@korn.s3an.me.uk) has joined #ceph
[12:16] * mjeanson (~mjeanson@bell.multivax.ca) has joined #ceph
[12:17] * Amto_res (~amto_res@ks312256.kimsufi.com) has joined #ceph
[12:17] * DrWhax (~whx@cryptohub.nl) has joined #ceph
[12:17] * ssejourne (~ssejourne@37.187.216.206) has joined #ceph
[12:22] * jtang_ (~jtang@80.111.83.231) Quit (Ping timeout: 480 seconds)
[12:24] * lucas1 (~Thunderbi@222.240.148.154) has joined #ceph
[12:25] * jtang_ (~jtang@80.111.83.231) has joined #ceph
[12:25] * jordanP (~jordan@185.23.92.11) has joined #ceph
[12:26] * jtang_ (~jtang@80.111.83.231) Quit (Remote host closed the connection)
[12:28] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[12:29] * lucas1 (~Thunderbi@222.240.148.154) Quit ()
[12:29] * jtang_ (~jtang@80.111.83.231) has joined #ceph
[12:33] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) has joined #ceph
[12:48] * imriz (~imriz@5.29.200.177) has joined #ceph
[12:55] * stephan (~stephan@62.217.45.26) has joined #ceph
[12:56] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[12:57] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[13:05] * JC (~JC@AMontpellier-651-1-32-204.w90-57.abo.wanadoo.fr) Quit (Quit: Leaving.)
[13:15] * Psi-Jack (~psi-jack@psi-jack.user.oftc.net) Quit (Quit: ZNC shutting down. I'll be back!)
[13:15] * Psi-Jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[13:15] * lincolnb (~lincoln@c-67-165-142-226.hsd1.il.comcast.net) Quit (Read error: Connection reset by peer)
[13:15] * cronix1 (~cronix@5.199.139.166) Quit (Ping timeout: 480 seconds)
[13:16] * cronix1 (~cronix@5.199.139.166) has joined #ceph
[13:19] * stephan (~stephan@62.217.45.26) Quit (Ping timeout: 480 seconds)
[13:20] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[13:20] * pressureman (~pressurem@62.217.45.26) Quit (Ping timeout: 480 seconds)
[13:20] * elder (~elder@c-24-245-18-91.hsd1.mn.comcast.net) has joined #ceph
[13:20] * ChanServ sets mode +o elder
[13:21] * lincolnb (~lincoln@c-67-165-142-226.hsd1.il.comcast.net) has joined #ceph
[13:21] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has joined #ceph
[13:23] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[13:23] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[13:26] * ikrstic_ (~ikrstic@93-86-144-171.dynamic.isp.telekom.rs) Quit (Ping timeout: 480 seconds)
[13:27] * vbellur (~vijay@121.244.87.117) Quit (Ping timeout: 480 seconds)
[13:28] * madkiss (~madkiss@ws14-141.sys11.net) has joined #ceph
[13:29] * stephan (~stephan@62.217.45.26) has joined #ceph
[13:30] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[13:30] * zidarsk8 (~zidar@2001:1470:fffd:101c:ea11:32ff:fe9a:870) has left #ceph
[13:39] * ikrstic_ (~ikrstic@79-101-227-243.dynamic.isp.telekom.rs) has joined #ceph
[13:41] * saurabh (~saurabh@121.244.87.117) Quit (Quit: Leaving)
[13:43] * vbellur (~vijay@121.244.87.124) has joined #ceph
[13:50] * shimo (~A13032@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Quit: shimo)
[13:56] <hedin> I have just tried to remove a HDD from a test cluster while running rados bench -p test 300 write and for about 4 minutes after the removal, all writes were halted and the logs started showing "slow request"... is it intentional that the cluster "pauses" for that long after a disk failure?
[13:57] * zhaochao (~zhaochao@123.151.134.238) has left #ceph
[14:00] * vbellur (~vijay@121.244.87.124) Quit (Quit: Leaving.)
[14:00] * pressureman (~pressurem@62.217.45.26) Quit (Ping timeout: 480 seconds)
[14:00] * stephan (~stephan@62.217.45.26) Quit (Ping timeout: 480 seconds)
[14:03] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[14:03] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[14:07] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) has joined #ceph
[14:09] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[14:09] * analbeard (~shw@185.28.167.198) Quit (Remote host closed the connection)
[14:09] * stephan (~stephan@62.217.45.26) has joined #ceph
[14:09] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[14:09] * analbeard (~shw@support.memset.com) has joined #ceph
[14:10] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[14:26] * rdas (~rdas@121.244.87.115) Quit (Quit: Leaving)
[14:29] * lalatenduM (~lalatendu@121.244.87.117) has joined #ceph
[14:32] * fejjerai (~quassel@corkblock.jefferai.org) Quit (Remote host closed the connection)
[14:35] * KevinPerks (~Adium@cpe-174-098-096-200.triad.res.rr.com) has joined #ceph
[14:37] * ganders (~root@200.0.230.235) has joined #ceph
[14:38] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) has joined #ceph
[14:40] * bkopilov (~bkopilov@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[14:44] * b0e (~aledermue@juniper1.netways.de) Quit (Quit: Leaving.)
[14:45] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[14:50] * via (~via@smtp2.matthewvia.info) Quit (Ping timeout: 480 seconds)
[14:50] * vmx (~vmx@p508A54DB.dip0.t-ipconnect.de) has joined #ceph
[14:52] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) Quit (Read error: Operation timed out)
[14:54] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) has joined #ceph
[14:55] * fejjerai (~quassel@corkblock.jefferai.org) has joined #ceph
[14:59] * fejjerai (~quassel@corkblock.jefferai.org) Quit (Remote host closed the connection)
[15:01] * vbellur (~vijay@122.167.82.113) has joined #ceph
[15:01] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has joined #ceph
[15:01] * ChanServ sets mode +v andreask
[15:03] * ScOut3R (~ScOut3R@catv-80-99-64-8.catv.broadband.hu) Quit (Read error: Operation timed out)
[15:04] * ikrstic_ (~ikrstic@79-101-227-243.dynamic.isp.telekom.rs) Quit (Quit: Konversation terminated!)
[15:12] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[15:12] * pressureman (~pressurem@62.217.45.26) Quit (Ping timeout: 480 seconds)
[15:12] * stephan (~stephan@62.217.45.26) Quit (Ping timeout: 480 seconds)
[15:12] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[15:13] * KevinPerks (~Adium@cpe-174-098-096-200.triad.res.rr.com) has left #ceph
[15:14] * bandrus (~Adium@c-4f665f1e-74736162.cust.telenor.se) has joined #ceph
[15:14] * KevinPerks (~Adium@cpe-174-098-096-200.triad.res.rr.com) has joined #ceph
[15:16] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[15:16] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[15:17] * pressureman (~pressurem@62.217.45.26) has joined #ceph
[15:20] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[15:21] * stephan (~stephan@62.217.45.26) has joined #ceph
[15:23] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[15:29] * shang (~ShangWu@175.41.48.77) Quit (Ping timeout: 480 seconds)
[15:35] * fejjerai (~quassel@corkblock.jefferai.org) has joined #ceph
[15:42] * AfC (~andrew@2001:44b8:31cb:d400:6e88:14ff:fe33:2a9c) Quit (Quit: Leaving.)
[15:42] * oro (~oro@2001:620:20:16:c1e4:f602:d5eb:19d6) has joined #ceph
[15:42] * t0rn (~ssullivan@2607:fad0:32:a02:d227:88ff:fe02:9896) has joined #ceph
[15:42] * sjm (~sjm@pool-72-76-115-220.nwrknj.fios.verizon.net) has joined #ceph
[15:50] * ikrstic (~ikrstic@77-46-238-93.static.isp.telekom.rs) has joined #ceph
[15:53] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[15:53] * shimo (~A13032@60.36.191.146) has joined #ceph
[15:54] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[15:54] * steveeJ (~junky@client156.amh.kn.studentenwohnheim-bw.de) has joined #ceph
[15:58] * drankis (~drankis__@89.111.13.198) Quit (Ping timeout: 480 seconds)
[15:58] * bandrus1 (~Adium@m90-141-180-55.cust.tele2.se) has joined #ceph
[15:59] * ikrstic (~ikrstic@77-46-238-93.static.isp.telekom.rs) Quit (Quit: Konversation terminated!)
[16:03] * bandrus (~Adium@c-4f665f1e-74736162.cust.telenor.se) Quit (Ping timeout: 480 seconds)
[16:04] * tdasilva (~quassel@nat-pool-bos-t.redhat.com) has joined #ceph
[16:04] * bandrus (~Adium@c-4f66d94b-74736162.cust.telenor.se) has joined #ceph
[16:04] * drankis (~drankis__@89.111.13.198) has joined #ceph
[16:06] * wschulze (~wschulze@cpe-69-206-251-158.nyc.res.rr.com) has joined #ceph
[16:09] * hyperbaba__ (~hyperbaba@private.neobee.net) Quit (Ping timeout: 480 seconds)
[16:09] * bandrus1 (~Adium@m90-141-180-55.cust.tele2.se) Quit (Ping timeout: 480 seconds)
[16:11] * primechu_ (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[16:11] * shang (~ShangWu@211.21.156.86) has joined #ceph
[16:12] * shimo (~A13032@60.36.191.146) Quit (Quit: shimo)
[16:18] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Ping timeout: 480 seconds)
[16:20] * markbby (~Adium@168.94.245.2) has joined #ceph
[16:25] * via (~via@smtp2.matthewvia.info) has joined #ceph
[16:27] * bandrus1 (~Adium@c-4f666096-74736162.cust.telenor.se) has joined #ceph
[16:28] * gregmark (~Adium@68.87.42.115) has joined #ceph
[16:28] * steveeJ (~junky@client156.amh.kn.studentenwohnheim-bw.de) Quit (Ping timeout: 480 seconds)
[16:29] * bandrus (~Adium@c-4f66d94b-74736162.cust.telenor.se) Quit (Ping timeout: 480 seconds)
[16:33] * drankis (~drankis__@89.111.13.198) Quit (Quit: Leaving)
[16:33] * drankis (~drankis__@89.111.13.198) has joined #ceph
[16:34] * i_m (~ivan.miro@gbibp9ph1--blueice4n1.emea.ibm.com) Quit (Quit: Leaving.)
[16:37] * dgarcia (~dgarcia@50-73-137-146-ip-static.hfc.comcastbusiness.net) has joined #ceph
[16:37] * JC (~JC@AMontpellier-651-1-32-204.w90-57.abo.wanadoo.fr) has joined #ceph
[16:40] <nyerup> tnt_, joao: The cluster is still working on deleting the snapshot. Just FYI.
[16:40] <tnt_> wow, that takes a while.
[16:40] <nyerup> Yeah.
[16:40] <nyerup> OSDs are on XFS.
[16:41] <nyerup> So I'm guessing it's basically a file system traversal.
[16:50] * theanalyst (~abhi@49.32.0.29) Quit (Remote host closed the connection)
[16:51] * angdraug (~angdraug@c-67-169-181-128.hsd1.ca.comcast.net) has joined #ceph
[16:57] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[16:57] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[16:58] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[16:59] * shimo (~A13032@60.36.191.146) has joined #ceph
[17:01] * xarses (~andreww@c-76-126-112-92.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[17:03] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[17:04] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[17:04] * kevinc (~kevinc__@client65-40.sdsc.edu) has joined #ceph
[17:07] * adamcrume (~quassel@2601:9:6680:47:6148:19de:d321:e04a) has joined #ceph
[17:09] * salgeras (~salgeras@sw4i.wifi.b92.net) Quit (Quit: Leaving)
[17:15] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[17:17] * zigo_ is now known as zigo
[17:17] * jlogan1 (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[17:18] * zack_dolby (~textual@p67f6b6.tokynt01.ap.so-net.ne.jp) has joined #ceph
[17:21] * hasues (~hasues@kwfw01.scrippsnetworksinteractive.com) has joined #ceph
[17:27] <tnt_> Anyone know how to specify a different user than 'admin' when using qemu ?
[17:27] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has joined #ceph
[17:30] * narb (~Jeff@38.99.52.10) has joined #ceph
[17:31] * shang (~ShangWu@211.21.156.86) Quit (Ping timeout: 480 seconds)
[17:31] * zidarsk8 (~zidar@89-212-142-10.dynamic.t-2.net) has left #ceph
[17:32] <tnt_> actually I need to set it n the config file somehow because I can't set it on the qemu line ...
[17:38] * KaZeR (~kazer@c-67-161-64-186.hsd1.ca.comcast.net) has joined #ceph
[17:39] * lalatenduM (~lalatendu@121.244.87.117) Quit (Quit: Leaving)
[17:42] * xarses (~andreww@12.164.168.117) has joined #ceph
[17:42] <tnt_> No one knows ? Damn, I sure hope it's possible ...
[17:45] * oro (~oro@2001:620:20:16:c1e4:f602:d5eb:19d6) Quit (Ping timeout: 480 seconds)
[17:45] * bandrus (~Adium@c-4f6667c3-74736162.cust.telenor.se) has joined #ceph
[17:48] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[17:48] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[17:48] * markbby (~Adium@168.94.245.2) has joined #ceph
[17:51] * \ask (~ask@oz.develooper.com) Quit (Quit: Bye)
[17:51] * bandrus1 (~Adium@c-4f666096-74736162.cust.telenor.se) Quit (Ping timeout: 480 seconds)
[17:52] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:52] * \ask (~ask@oz.develooper.com) has joined #ceph
[17:52] * madkiss (~madkiss@ws14-141.sys11.net) Quit (Quit: Leaving.)
[17:53] * dtibarra (~davidi@192.185.1.20) has joined #ceph
[17:53] * bandrus1 (~Adium@c-4f666e7c-74736162.cust.telenor.se) has joined #ceph
[17:55] * kevinc (~kevinc__@client65-40.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[17:57] <hedin> I get "Invalied argument" on 7 of 9 OSD's when I try to start them but I ca't see the error:s https://dpaste.de/z4iS
[17:58] * kevinc (~kevinc__@client65-40.sdsc.edu) has joined #ceph
[17:59] * bandrus (~Adium@c-4f6667c3-74736162.cust.telenor.se) Quit (Ping timeout: 480 seconds)
[18:01] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[18:01] * markbby (~Adium@168.94.245.2) has joined #ceph
[18:06] * yuriw (~Adium@c-76-126-35-111.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[18:07] * yuriw1 (~Adium@c-76-126-35-111.hsd1.ca.comcast.net) has joined #ceph
[18:10] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Quit: Leaving.)
[18:16] * adamcrume (~quassel@2601:9:6680:47:6148:19de:d321:e04a) Quit (Remote host closed the connection)
[18:18] * analbeard (~shw@support.memset.com) Quit (Quit: Leaving.)
[18:25] * _are_ (~quassel@h1417489.stratoserver.net) Quit (Ping timeout: 480 seconds)
[18:26] * lpabon_test (~quassel@nat-pool-bos-u.redhat.com) has joined #ceph
[18:27] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[18:35] * ircolle (~Adium@h96-61-157-61.mtjltn.dsl.dynamic.tds.net) has joined #ceph
[18:35] * ircolle (~Adium@h96-61-157-61.mtjltn.dsl.dynamic.tds.net) Quit ()
[18:36] * marrusl (~mark@faun.canonical.com) Quit (Quit: sync && halt)
[18:37] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[18:37] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[18:37] * adamcrume (~quassel@50.247.81.99) has joined #ceph
[18:38] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[18:41] * JayJ (~jayj@157.130.21.226) has joined #ceph
[18:45] * nljmo (~nljmo@5ED6C263.cm-7-7d.dynamic.ziggo.nl) Quit (Quit: Textual IRC Client: www.textualapp.com)
[18:45] * Discard (~discard@213-245-29-151.rev.numericable.fr) has left #ceph
[18:47] * kevinc (~kevinc__@client65-40.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[18:49] * JayJ (~jayj@157.130.21.226) Quit (Ping timeout: 480 seconds)
[18:50] * michalefty (~micha@188-195-129-145-dynip.superkabel.de) Quit (Quit: Leaving.)
[18:51] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) has joined #ceph
[18:53] * lpabon (~lpabon@nat-pool-bos-t.redhat.com) Quit (Quit: ZNC - http://znc.in)
[18:53] * lcavassa (~lcavassa@78.25.240.221) Quit (Remote host closed the connection)
[18:54] * lpabon (~lpabon@nat-pool-bos-t.redhat.com) has joined #ceph
[18:54] * wayneeseguin (sid2139@id-2139.uxbridge.irccloud.com) has joined #ceph
[18:54] * reed (~reed@75-101-54-131.dsl.static.sonic.net) has joined #ceph
[18:55] * lpabon_test (~quassel@nat-pool-bos-u.redhat.com) Quit (Remote host closed the connection)
[18:55] * adamcrume (~quassel@50.247.81.99) Quit (Read error: Connection reset by peer)
[18:56] * lpabon_test (~quassel@nat-pool-bos-u.redhat.com) has joined #ceph
[18:57] * kevinc (~kevinc__@client65-40.sdsc.edu) has joined #ceph
[18:58] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[19:00] <lpabon_test> lpabon: test
[19:01] * lpabon_test (~quassel@nat-pool-bos-u.redhat.com) Quit (Remote host closed the connection)
[19:03] * drankis_ (~drankis__@89.111.13.198) has joined #ceph
[19:04] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Read error: Connection reset by peer)
[19:04] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[19:07] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (Ping timeout: 480 seconds)
[19:10] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[19:10] * rweeks (~rweeks@pat.hitachigst.com) has joined #ceph
[19:12] <joshd> tnt_: on the command line pass id=clientname
[19:13] <tnt_> joshd: yes, but I'm having trouble getting that special arg through the xen layer ... (to reach the qdisk driver).
[19:14] <tnt_> so I was hoping for an option in the conig file.
[19:14] <tnt_> but looking at the source, there is none for this ... it's hardcoded to 'admin' by default.
[19:14] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) has joined #ceph
[19:16] <joshd> tnt_: setting CEPH_ARGS="--id ..." may work, if the environment is preserved when qemu is run
[19:22] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Remote host closed the connection)
[19:23] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[19:23] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) Quit (Quit: Leaving.)
[19:23] * adamcrume (~quassel@2601:9:6680:47:c996:ca64:72ad:eb72) has joined #ceph
[19:24] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (Ping timeout: 480 seconds)
[19:25] * koleosfuscus (~koleosfus@adsl-84-226-68-69.adslplus.ch) has joined #ceph
[19:26] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[19:26] * jordanP (~jordan@185.23.92.11) Quit (Quit: Leaving)
[19:26] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) has joined #ceph
[19:27] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[19:29] * Nacer_ (~Nacer@252-87-190-213.intermediasud.com) Quit (Read error: Operation timed out)
[19:29] <tnt_> joshd: nm, I finally figured out how to pass that through the xen layer correctly :)
[19:29] * sjusthm (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) Quit (Read error: Operation timed out)
[19:32] * ron-slc (~Ron@173-165-129-125-utah.hfc.comcastbusiness.net) Quit (Quit: Leaving)
[19:32] * zerick (~eocrospom@190.187.21.53) has joined #ceph
[19:32] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Ping timeout: 480 seconds)
[19:33] * Cube (~Cube@66.87.67.122) has joined #ceph
[19:37] * sjustlaptop (~sam@172.56.16.229) has joined #ceph
[19:38] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (Ping timeout: 480 seconds)
[19:41] <lupu1> hello, I am trying to apply ceph patches to openstack - icehouse (for cloning and live migration)
[19:42] <lupu1> is there another repo besides the one of: angdraug ( https://github.com/angdraug/nova/commits/rbd-ephemeral-clone-stable-icehouse )
[19:43] * koleosfuscus (~koleosfus@adsl-84-226-68-69.adslplus.ch) Quit (Quit: koleosfuscus)
[19:43] * ron-slc (~Ron@173-165-129-125-utah.hfc.comcastbusiness.net) has joined #ceph
[19:45] <lupu1> or is that the "official" way ?
[19:46] * bkopilov (~bkopilov@213.57.17.88) has joined #ceph
[19:46] <angdraug> the official way is to push for merging all these patches for juno :)
[19:47] <angdraug> which I'm trying to do in https://review.openstack.org/102064 and linked reviews
[19:48] * michalefty (~micha@188-195-129-145-dynip.superkabel.de) has joined #ceph
[19:48] <angdraug> until that's all merged, I think my repo is the most comprehensive collection of ceph related fixes for nova
[19:48] * shang (~ShangWu@220-135-203-169.HINET-IP.hinet.net) Quit (Ping timeout: 480 seconds)
[19:49] <angdraug> there's also joshd's repo: https://github.com/jdurgin/nova/
[19:49] <angdraug> but I tried to pull everything from there into my branch
[19:49] <joshd> use angdraug's, it's more up to date
[19:49] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[19:52] <lupu1> is it ok to replace the whole /usr/lib/python2.7/dist-packages/nova with yours ?
[19:53] <lupu1> I am on Ubuntu 14.04 and have installed icehouse from apt
[19:54] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) Quit (Quit: Leaving)
[19:54] <angdraug> yes
[19:55] <angdraug> it's based on 2014.1.1 tag
[19:56] <lupu1> ok, i will try patching
[19:57] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (Ping timeout: 480 seconds)
[19:58] * dtibarra (~davidi@192.185.1.20) Quit (Remote host closed the connection)
[19:59] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) has joined #ceph
[20:00] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) has joined #ceph
[20:00] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[20:00] * vmx (~vmx@p508A54DB.dip0.t-ipconnect.de) Quit (Quit: Leaving)
[20:02] <angdraug> a small correction
[20:03] <angdraug> not sure where you get your packages from, but normally you want to replace /usr/share/pyshared/nova, not /usr/lib/...
[20:09] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[20:09] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) Quit (Quit: valeech)
[20:10] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[20:11] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[20:12] * vilobhmm (~vilobhmm@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) has joined #ceph
[20:12] <vilobhmm> hi nhm
[20:12] <vilobhmm> is ceph locality aware ?
[20:15] * drankis_ (~drankis__@89.111.13.198) Quit (Ping timeout: 480 seconds)
[20:16] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (Ping timeout: 480 seconds)
[20:17] <vilobhmm> sage, dmick : ping
[20:18] <dmick> vilobhmm: how can I help you?
[20:18] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[20:20] * zerick (~eocrospom@190.187.21.53) Quit (Remote host closed the connection)
[20:20] * rotbeard (~redbeard@b2b-94-79-138-170.unitymedia.biz) Quit (Quit: Leaving)
[20:28] * michalefty (~micha@188-195-129-145-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[20:35] * sjustlaptop (~sam@172.56.16.229) Quit (Ping timeout: 480 seconds)
[20:44] * rendar (~I@95.238.179.240) Quit (Ping timeout: 480 seconds)
[20:44] * michalefty (~micha@188-195-129-145-dynip.superkabel.de) has joined #ceph
[20:46] * rendar (~I@95.238.179.240) has joined #ceph
[20:47] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[20:57] * kevinc (~kevinc__@client65-40.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[20:57] * kevinc (~kevinc__@client65-40.sdsc.edu) has joined #ceph
[20:59] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) Quit (Quit: Leaving.)
[21:02] * Nacer_ (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[21:02] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Read error: Connection reset by peer)
[21:03] * sjustlaptop (~sam@172.56.16.229) has joined #ceph
[21:04] * bandrus1 (~Adium@c-4f666e7c-74736162.cust.telenor.se) Quit (Quit: Leaving.)
[21:06] <vilobhmm> dmick : is ceph locality aware ?
[21:06] * rotbeard (~redbeard@aftr-37-24-147-15.unity-media.net) has joined #ceph
[21:07] * ron-slc (~Ron@173-165-129-125-utah.hfc.comcastbusiness.net) Quit (Remote host closed the connection)
[21:11] * angdraug (~angdraug@c-67-169-181-128.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[21:11] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) has joined #ceph
[21:12] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) has joined #ceph
[21:17] * michalefty (~micha@188-195-129-145-dynip.superkabel.de) Quit (Quit: Leaving.)
[21:21] * sjustlaptop (~sam@172.56.16.229) Quit (Ping timeout: 480 seconds)
[21:22] * sjustlaptop (~sam@172.56.40.20) has joined #ceph
[21:24] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) has joined #ceph
[21:24] <dmick> vilobhmm: I'm not sure what you mean; are you talking about I18N/L10N/message catalogs? if so, no, but there was a CDS discussion about that a couple of weeks ago
[21:24] <dmick> (although really it was about Calamari, not Ceph)
[21:25] <dmick> but maybe that's not what you mean
[21:25] * valeech (~valeech@pool-71-171-123-210.clppva.fios.verizon.net) Quit ()
[21:44] * kevinc (~kevinc__@client65-40.sdsc.edu) Quit (Quit: This computer has gone to sleep)
[21:46] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[21:46] * angdraug (~angdraug@12.164.168.117) has joined #ceph
[21:49] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[21:54] * sjustlaptop1 (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) has joined #ceph
[21:56] * sjustlaptop (~sam@172.56.40.20) Quit (Ping timeout: 480 seconds)
[21:58] * sjustlaptop1 (~sam@66-214-251-229.dhcp.gldl.ca.charter.com) Quit ()
[21:58] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has joined #ceph
[21:58] * ChanServ sets mode +v andreask
[22:00] <vilobhmm> dmick : I mean locality in terms of data placement
[22:03] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has left #ceph
[22:04] * rwheeler (~rwheeler@nat-pool-bos-u.redhat.com) has joined #ceph
[22:05] <dmick> it would be good to elaborate on your question, I think
[22:06] * Sysadmin88 (~IceChat77@94.4.20.0) has joined #ceph
[22:11] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) Quit (Quit: Leaving.)
[22:12] <iggy> local reads if possible?
[22:15] * mrjack_ (mrjack@office.smart-weblications.net) has joined #ceph
[22:16] * diegows (~diegows@190.190.5.238) has joined #ceph
[22:19] * adamcrume (~quassel@2601:9:6680:47:c996:ca64:72ad:eb72) Quit (Remote host closed the connection)
[22:22] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[22:29] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Quit: Leaving.)
[22:36] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[22:41] * zerick (~eocrospom@190.187.21.53) has joined #ceph
[22:42] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) Quit (Quit: Leaving.)
[22:45] <imriz> iggy, not as I know. It will always go to the leader of the PG
[22:46] <imriz> in terms of writing objects, it will spread the data according to the crush rules
[22:46] <imriz> which can separate the data based on hosts, racks, datacenters etc.
[22:47] <imriz> I've heard that there is an ongoing work to allow reads from secondary osds in a pg, but I am not sure
[22:48] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) Quit (Quit: Leaving.)
[22:49] <vilobhmm> okay imriz
[22:49] <imriz> your use case includes low latency?
[22:49] <imriz> is that why you ask about locality?
[22:49] <vilobhmm> imriz : yes low latency
[22:49] * t0rn (~ssullivan@2607:fad0:32:a02:d227:88ff:fe02:9896) Quit (Quit: Leaving.)
[22:50] <imriz> small objects?
[22:50] <vilobhmm> if there is some data placement locality we can get low latency
[22:50] <imriz> like few K's?
[22:50] <vilobhmm> yes considering RBD where we will have small objects
[22:50] <vilobhmm> yes
[22:50] <vilobhmm> iggy, imriz: i dont think we can modify the crush rules, can we ?
[22:50] <imriz> what is the latency you're seeing right now?
[22:50] <imriz> yes you can
[22:51] <imriz> but I don't think that will help you much right now
[22:51] <imriz> what is the latency you're seeing?
[22:51] <imriz> I have a workload with small (4-5kb~ objects) with the radosgw
[22:52] <imriz> and I was seeing high latency as well
[22:52] <vilobhmm> for 16 concurrent 4k writes
[22:52] <vilobhmm> we see 100 millsecs
[22:52] <imriz> writes
[22:52] <imriz> or reads?
[22:52] <vilobhmm> 125
[22:52] <vilobhmm> writes
[22:52] <vilobhmm> 125 millisecs
[22:52] * ganders (~root@200.0.230.235) Quit (Quit: WeeChat 0.4.1)
[22:52] <imriz> which version?
[22:52] <vilobhmm> so we can manipulate crush rules and control the data placement
[22:52] <vilobhmm> 0.72
[22:53] * Nacer_ (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) Quit (Remote host closed the connection)
[22:53] * joshd (~jdurgin@2607:f298:a:607:ac96:4f87:31e:c2d9) Quit (Ping timeout: 480 seconds)
[22:54] <imriz> what are the values of "filestore_max_inline_xattr_size" and "filestore_max_inline_xattrs"?
[22:55] <imriz> I also found this article to be very helpful in tuning: http://ceph.com/community/ceph-bobtail-jbod-performance-tuning/
[22:55] <imriz> it's bobtail, but nevertheless helpful
[22:56] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) has joined #ceph
[22:57] <imriz> you can try, at your own risk, to set filestore_max_inline_xattr_size=65536 and filestore_max_inline_xattrs = 10
[22:58] <imriz> it was a tip I got to lower the usage of the leveldb
[22:58] <imriz> it helped in my setup
[22:58] <imriz> but cannot assure you how it will work in yours :)
[23:00] * sage (~quassel@cpe-23-242-158-79.socal.res.rr.com) Quit (Read error: Operation timed out)
[23:00] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) has joined #ceph
[23:03] * joshd (~jdurgin@2607:f298:a:607:fd80:ba24:7de3:bb37) has joined #ceph
[23:03] <vilobhmm> sure
[23:03] <vilobhmm> thanks imriz
[23:04] <imriz> np
[23:05] <vilobhmm> imriz : regarding modifying the crush rules; i was under the assumption that crush rules which are governmned by the crush algorithm can???t be tuned
[23:05] <imriz> read this: http://ceph.com/docs/master/rados/operations/crush-map/
[23:05] <vilobhmm> tuning the crush rules is this documented somewhere i couldnt find any rlevant
[23:05] <vilobhmm> ok cool
[23:05] <vilobhmm> thanks
[23:06] <imriz> but I really think you should avoid it unless you know what you're doing
[23:07] * sage (~quassel@cpe-23-242-158-79.socal.res.rr.com) has joined #ceph
[23:07] * ChanServ sets mode +o sage
[23:07] * aldavud (~aldavud@217-162-119-191.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[23:08] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) Quit (Quit: Leaving)
[23:15] <vilobhmm> sure
[23:17] * Nacer (~Nacer@c2s31-2-83-152-89-219.fbx.proxad.net) has joined #ceph
[23:18] * analbeard (~shw@host86-155-107-195.range86-155.btcentralplus.com) has joined #ceph
[23:22] * Tamil (~Adium@cpe-108-184-74-11.socal.res.rr.com) has joined #ceph
[23:24] * aarcane (~aarcane@99-42-64-118.lightspeed.irvnca.sbcglobal.net) has joined #ceph
[23:32] * imjustmatthew (~imjustmat@pool-74-110-226-158.rcmdva.fios.verizon.net) has joined #ceph
[23:35] * primechu_ (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[23:36] * rwheeler (~rwheeler@nat-pool-bos-u.redhat.com) Quit (Quit: Leaving)
[23:37] * primechuck (~primechuc@host-95-2-129.infobunker.com) has joined #ceph
[23:38] * analbeard (~shw@host86-155-107-195.range86-155.btcentralplus.com) Quit (Quit: Leaving.)
[23:40] * primechuck (~primechuc@host-95-2-129.infobunker.com) Quit (Remote host closed the connection)
[23:40] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) has joined #ceph
[23:40] * davidzlap (~Adium@ip68-4-173-198.oc.oc.cox.net) Quit ()
[23:41] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[23:49] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has joined #ceph
[23:49] * ChanServ sets mode +v andreask
[23:50] * madkiss (~madkiss@p5795B96C.dip0.t-ipconnect.de) Quit (Quit: Leaving.)
[23:51] * andreask (~andreask@h081217017238.dyn.cm.kabsi.at) has left #ceph
[23:52] * AfC (~andrew@2001:44b8:31cb:d400:6e88:14ff:fe33:2a9c) has joined #ceph
[23:52] * kevinc (~kevinc__@client65-40.sdsc.edu) has joined #ceph
[23:53] * rendar (~I@95.238.179.240) Quit ()
[23:55] * nljmo (~nljmo@5ED6C263.cm-7-7d.dynamic.ziggo.nl) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.