#ceph IRC Log


IRC Log for 2011-05-05

Timestamps are in GMT/BST.

[0:00] <Tv> it wasn't meant for direct human input
[0:00] <Tv> but it is flexible, extensible, and fully machine processable
[0:00] <cmccabe> tv: to me, markup language implies something a human can write
[0:00] <Tv> to me it implies something that can add markup to mostly textual content
[0:00] <cmccabe> tv: ok, then MS Word format is a "markup language"
[0:01] <cmccabe> tv: this is a little bit silly, no?
[0:01] <Tv> xml with a good editor is roughly comparable to using WordPerfect, from back in the day
[0:02] <cmccabe> tv: XML is a solution looking for a problem
[0:02] <Tv> trollface: bleh that message hides the underlying error
[0:02] <Tv> trollface: any change something else printed it, or if that's direct from your code, can you capture the return value?
[0:02] <cmccabe> tv: just using JSON with some kind of schema would be so much better
[0:02] * trollface is using xml to interop with dudes who write their stuff in an obscure language of their choice. So I do http requests to them, get back xml, parse it, and then use the protobuf-based rpcs within my system
[0:02] <trollface> Tv: it's from rbd create :(
[0:02] * cephuser1 (~cephuser1@173-24-225-53.client.mchsi.com) Quit (Remote host closed the connection)
[0:03] <cmccabe> trollface: XML is still better than the GDF format
[0:03] <trollface> [root@mpi-m9 ~]# rbd create foo --size 100
[0:03] <trollface> 2011-05-05 01:56:38.999735 7f9221f01740 librbd: failed to assign a block name for image
[0:03] <trollface> create error: Input/output error
[0:03] <Tv> trollface: ah that's what i wanted
[0:03] <trollface> is it monitor or osd error? I can pump some debug into it
[0:03] <cmccabe> trollface: GDF was another "interop" format designed to be parsable by anything...
[0:04] <cmccabe> trollface: where "anything" also included COBOL
[0:04] <trollface> (I also had some fun inserting stripped libcls_rbd.so but that's another story)
[0:04] <Tv> what's CLS_LOG?
[0:04] <cmccabe> trollface: so every line was 80 characters long, and fields were given at fixed intervals
[0:04] <trollface> cmccabe: well... in my case, it works okay
[0:04] <trollface> cmccabe: I can parse it both from python and from scala
[0:05] <cmccabe> trollface: I'm just having a really hard time with XML lately
[0:05] <Tv> ah that just goes to dout
[0:05] <cmccabe> trollface: some things are picky about namespaces, some aren't
[0:05] <cmccabe> trollface: amazon rejects XML that looks the same as other XML with a bland "something failed, somewhere, good luck!"
[0:06] <Tv> trollface: so if you see no other error messages, then that means cls_cxx_read returned -EIO.. digging..
[0:06] <Tv> cmccabe: not doing namespaces right is just lazy programming
[0:06] <trollface> bah, it's simple enough here. nobody cares about namespaces, it's simple <obj attr=val><subobj attr=val>blah</subobj>...
[0:07] <cmccabe> trollface: well, lxml won't let me ignore namespaces
[0:07] <trollface> if it's parseable (usually means sticking <?... utf8?>
[0:07] <cmccabe> trollface: and I don't think any of the other python libraries will either...
[0:07] <Tv> i wish James Clark had won that argument
[0:07] <cmccabe> trollface: actually, amazon doesn't seem to care about the xml version thing
[0:08] <Tv> xml elements with namespaces would look like <{http://example.com/foo}elem ...>
[0:08] <Tv> there'd be no way to avoid doing them right
[0:08] <Tv> instead of the whole xmlns thing, where if you use a low-level access lib, you need to handle namespace logic yourself
[0:08] <trollface> cmccabe: I use ElementTree.fromstring
[0:09] <trollface> cmccabe: but I see your point, your usecase is quite different
[0:09] <cmccabe> tv: ElementTree is provided by a ton of libraries
[0:09] <cmccabe> er trollface
[0:09] <cmccabe> trollface: lxml, cElementTree, and ElementTree all provide a class ElementTree
[0:09] <cmccabe> trollface: but if you use the cElementTree or ElementTree library, it sticks "ns0" in front of absolutely everything
[0:10] <Tv> trollface: it seems your error is basically getting EIO from talking to the osds; that is pretty much the "default error" with ceph, and can mean just about anything from timeouts onwards.. what does "ceph health" say
[0:10] <cmccabe> trollface: and online there's a really annoying guy who vigilantly flames anyone who suggests this should be configurable by pointing out that this behavior is allowed by the standard
[0:10] <trollface> cmccabe: you're probably right
[0:10] <Tv> cmccabe: please use lxml, others are nowhere near as good
[0:10] <trollface> Tv: HEALTH_OK
[0:11] <trollface> Tv: also, filesystem interface works
[0:11] <trollface> I have rsync in other window writing to this cluster
[0:11] <trollface> it's just something wrong with rbd, I probably broke something or didn't setup correctly
[0:11] <trollface> I added class and activated it
[0:12] <Tv> trollface: yeah it's talking to the class..
[0:12] <Tv> trollface: the error comes from the rbd class submitting osd operations
[0:12] <Tv> function cls_cxx_read
[0:12] <Tv> that's my best read of it
[0:12] <trollface> rbd (v1.3 [x86-64]) [active]
[0:12] <Tv> i'm sorry, i don't really have much more to suggest
[0:13] <trollface> bah... the classes are inside monitor, right
[0:13] <Tv> i don't see josh on irc right now, he might have a better clue
[0:13] <Tv> they're kinda everywhere
[0:13] <Tv> that's the point of them
[0:13] <Tv> if something went wrong with loading the class everywhere, i'm not sure what would have happened..
[0:14] <Tv> but that might also explain the error, not sure
[0:14] <cmccabe> I thought we were getting rid of class loading
[0:14] <cmccabe> did sage make that change yet?
[0:14] <Tv> cmccabe: class distribution yes, loading no
[0:14] <trollface> that's funnyI guess I have to disable stripping and recompile this thing
[0:14] <Tv> that's my understanding of it
[0:17] <cmccabe> so I just encountered another frustrating "maybe-error" on rgw
[0:17] <cmccabe> tv: bucket = get_new_bucket()
[0:17] <cmccabe> key = bucket.new_key('foo')
[0:17] <cmccabe> key.set_contents_from_string('bar')
[0:17] <cmccabe> key.set_xml_acl(XML_1)
[0:17] <cmccabe> tv: what do you think should happen?
[0:17] <cmccabe> tv: apparently sometimes set_xml_acl fails with 404, NoSuchKey
[0:18] <Tv> cmccabe: screwed up caching?
[0:18] <cmccabe> tv: well, first of all, is this allowed to fail?
[0:19] <cmccabe> tv: according to a strict reading of the spec, this failure might be something allowed by eventual consistency
[0:19] <Tv> that seems like aws's "read-after-write" guarantee would mean that must work
[0:19] <trollface> okay, I guess I'll continue my pointless quest tomorrow :(
[0:19] <cmccabe> tv: at least that was my first thought
[0:19] <trollface> cu later
[0:19] <cmccabe> trollface: sorry we couldn't find the root cause today
[0:19] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[0:19] <Tv> cmccabe: i think it goes like this; most of AWS docs are written before the read-after-write stuff came up; the spec is mostly silent about this
[0:19] <cmccabe> trollface: greg or sage might have some info about that class-loading stuff
[0:20] <cmccabe> tv: well, have we decided what kind of consistency rgw should have?
[0:20] <Tv> cmccabe: aws implementation routed write actions to master copy of data, others saw possibly laggy read-only replicas
[0:20] <cmccabe> tv: I heard arguments for both read-after-write and eventual consistency, but I don't know who actually won that argument
[0:20] <Tv> cmccabe: huge customer demand made them add "read-after-write", that is cache invalidation, to you never see stale reads
[0:21] <Tv> cmccabe: that transition is complete for only some regions
[0:21] <cmccabe> I'm going to file this as a bug and you guys can debate it
[0:21] <Tv> cmccabe: they are definitely moving toward read-after-write, and that's what we should implement (that's already been agreed on)
[0:21] <Tv> cmccabe: but in this case, it seems our request routing is 100% random, that is, worse than aws ever was
[0:21] <Tv> cmccabe: or something
[0:22] <Tv> i don't actually know where if anywhere rgw has caching
[0:22] <Tv> or whether it touches non-primary replicas ever
[0:22] <Tv> cmccabe: let me explicit about this, one more time: we've already decided to go for "read-after-write"
[0:22] <Tv> the implementation is probably not up to it, though
[0:23] <cmccabe> tv: got it
[0:23] <Tv> and that sort of stuff is very hard to test with integration tests
[0:23] <Tv> apart from just stressing it & hoping
[0:23] <Tv> and rgw has no unit tests
[0:23] <Tv> so.. meh
[0:23] <Tv> if you file ticket, please copy-paste the above
[0:24] <cmccabe> do we have an umbrella issue for consistency problems?
[0:24] <Tv> as this seems to never be clear enough
[0:24] <Tv> i don't know of such
[0:24] <Tv> i've really looked at rgw code very little
[0:24] <Tv> i got s3-tests to mostly work against aws, ran against rgw, flagged specific reproducible problems, filed tickets, and moved on
[0:25] <Tv> at that time there were enough problems that more work on s3-tests seemed pointless
[0:28] <gregaf1> there's no caching in rgw
[0:28] <Tv> well, we need some reason why create-object & set-acl makes set-acl not see that the object is there
[0:29] <Tv> cmccabe: ^
[0:29] <gregaf1> I'm a bit concerned about race conditions in terms of competing writes but whether that's actually a problem depends on how it's implemented, I'm not sure
[0:29] <Tv> gregaf1: yeah it needs some kind of locking or a well-defined last-write-wins strategy
[0:30] <Tv> with the right unit of atomicity for whatever is appropriate
[0:30] <gregaf1> huh, I actually don't know how create-object and set-acl would think there's not an object for the set-acl to run on...
[0:30] <gregaf1> like, rgw doesn't return until it's completed those writes to the cluster I thought
[0:30] <Tv> cmccabe: perhaps this is more about xml acl handling bugs
[0:31] <Tv> cmccabe: i only tested json stuff (boto's default)
[0:31] <Tv> cmccabe: if you time.sleep(5) in the middle, does it still fail like that, etc
[0:32] <cmccabe> tv: I only got #1060 to happen once
[0:32] <Tv> yay for races
[0:33] <Tv> (you wanted a pony, you got a whole race track!)
[0:34] <cmccabe> also the bug tracker decided it would be fun to create #1061 as an exact copy of #1060 even though I only pressed the button once
[0:34] <cmccabe> attempts to close #1061, add comments to #1061, or do anything at all with 1061 result in internal database errors
[0:34] <Tv> hahaha
[0:35] <cmccabe> seems like the racetrack is bigger than I thought
[0:35] <Tv> it has rails
[0:42] <cmccabe> solving each bug today requires solving another bug
[0:42] <cmccabe> so that I never actually make any progress
[0:42] <cmccabe> just infinite recursion
[0:44] <Tv> fortunately, it is impossible for finite source to contain infinite bugs, so all you need to do is to introduce fewer bugs than you fix ;)
[0:44] <Tv> you'll dig yourself out of the hole some day
[0:45] <cmccabe> well, yehuda's not writing any new rgw code
[0:46] <gregaf1> now you know how I feel every time I work on the MDS :p
[0:46] <cmccabe> but I have to write at least a little bit to fix the obvious things
[0:46] <Tv> and this is why i'm so vehement about unit tests
[0:46] <Tv> they're a good ladder to climb out of the hole with
[0:48] <cmccabe> tv: unit tests are good, but integration tests are also pretty important
[0:49] <Tv> yeah, different purposes, one can't completely cover the role of the other
[0:49] <cmccabe> tv: we would make a lot more progress if there was someone else responsible for testing this stuff
[0:49] <Tv> something like.. unit tests make development easier, integration tests make support & qa easier
[0:49] <cmccabe> tv: as it is, the more time I spend testing, the less productive I "appear" to be
[0:49] <Tv> cmccabe: fwiw, i do not agree with the idea of qa being separate
[0:49] <cmccabe> tv: also I'm primarily a developer so testing is not really my focus
[0:50] <cmccabe> tv: this may be controversial, but I also think that unit tests are more important in dynamically typed languages than in statically typed ones
[0:50] <Tv> sure, somebody could be in charge of regularly running things and gathering results
[0:50] <cmccabe> tv: the kernel ,for example, has no unit tests
[0:50] <Tv> but unless you work with the code, you can't debug it
[0:50] <cmccabe> tv: despite being a fairly large and complex piece of software
[0:50] <Tv> cmccabe: kernels are very different from userspace code
[0:51] <Tv> cmccabe: it's very hard to run most kernel code in userspace; i've done that, to test it better, it's not fun
[0:51] <cmccabe> tv: I feel like a lot of the supposed productivity gains of dynamic typing are actually eaten up by the need to write more unit tests
[0:51] <Tv> not getting into that fight
[0:51] <Tv> (static typing is no excuse for sloppy)
[0:51] <cmccabe> tv: I'm just raising the question
[0:52] <Tv> static typing says *nothing* about what your code actually does
[0:52] <Tv> only that you didn't make certain classes of simple errors
[0:52] <Tv> most python unit tests don't even try to cover those errors
[0:52] <Tv> the thinking is, if caller does that, it's often pretty simple to figure out
[0:52] <cmccabe> tv: I'm just annoyed that every time I change anything, it may cause a runtime error to pop up somewhere else
[0:52] <Tv> you're doing it wrong
[0:53] <Tv> ;)
[0:53] <cmccabe> tv: I have to exercise literally every code path to be sure that won't happen
[0:53] <cmccabe> tv: that's why I wrote test-obsync.py
[0:53] <Tv> cmccabe: you. have. to. do. that. anyway.
[0:53] <Tv> not running every combo in unit tests means you didn't test everything.
[0:53] <cmccabe> tv: ... unless you're something like the linux kernel, that code is way trivial
[0:53] <cmccabe> tv: <-- sarcasm
[0:54] <cmccabe> tv: anyway, I'm sorry. I didn't mean to have the static-vs-dynamic flamewar
[0:54] <Tv> almost all the screwups you see in C are about error handling paths
[0:54] <Tv> if you don't test them, you're doing the easy parts only
[0:54] <Tv> guess whether that's sufficient
[0:54] <cmccabe> yeah, those paths are often buggy, and that is a problem
[0:55] <cmccabe> there was some talk about a general fault injection framework for the kernel, but I don't know if it ever came to fruition
[0:55] <Tv> oh they have that
[0:55] <Tv> e.g. make x% of memory allocations fail
[0:55] <Tv> it's just not run in "unit tests", but in integration tests
[0:56] <Tv> as they don't have good, isolated, units
[0:56] <cmccabe> well, the kernel does have good isolated units
[0:56] <cmccabe> things like kernel modules
[0:56] <cmccabe> actually quite a lot of effort is spent on making things modular
[0:58] <Tv> the modules are not isolated from the core
[0:58] <Tv> they require a bunch of functions that are incredible painful to reimplement, just to run that C in userspace
[0:58] <Tv> really, i did that, for big chunks of the firewall product
[0:59] <cmccabe> you are confusing isolation with lack of dependencies
[0:59] <Tv> it was not fun, in general it was only doable for "purely algorithmic" subcomponents, and even then i needed to kludge a lot to make it appear it has direct access to kernels data structures
[0:59] <Tv> you're confusing modular with isolated ;)
[0:59] <cmccabe> I guess one thing that is true is that simulating hardware is a pain
[0:59] <Tv> it's not about the hardware, it's about stuff like struct sk_buff
[1:00] <cmccabe> why not just include that header and use it?
[1:00] <cmccabe> I mean people do copy kernel headers all the time
[1:00] <Tv> because the header depends on a bazillion things about the kernel
[1:01] <Tv> if you mean the glibc interface, that's a tiny fraction of the size of the in-kernel apis
[1:01] <Tv> and explicitly built not to expose internals
[1:01] <cmccabe> I mean look at some of our unit tests.
[1:01] <cmccabe> they depend on "a bazillion things about ceph"
[1:02] <Tv> sometimes, i'm not sure why i have these conversations
[1:02] <Tv> kernels are very different
[1:02] <cmccabe> like ceph::Lock, ceph::bufferlist, utime_t
[1:02] <cmccabe> tv: I respect your experience, I'm just trying to understand
[1:03] <Tv> cmccabe: imagine not being able to include the header for that ceph::Lock, because it does things that don't exist in userspace
[1:03] <Tv> cmccabe: now imagine that for every single core Ceph mechanism, locks messenger bufferlist etc
[1:04] <cmccabe> tv: it really seems like you might have been better off running the unit tests in kernel space under UML or something
[1:05] <Tv> which is roughly what they're doing, except it's very hard to isolate the units, so they don't try
[1:05] <Tv> so they run a full kernel, with error injection etc as wanted, and then run workloads against them
[1:05] <Tv> but that's integration testing, because the units are so huge
[1:06] <cmccabe> tv: to me it seems like any large piece of software will have a lot of the same issues
[1:06] <cmccabe> tv: can't run the foo module without the bar, which in turn requires the baz module
[1:06] <Tv> i will give you that C is hard to unit test, because it's not very dynamic
[1:06] <cmccabe> tv: supposedly you should solve this with "mockups", but that just chews developer time
[1:06] <Tv> but the kernel is special because it's all a single executable
[1:06] <Tv> so C symbol clashes etc prevent you from "loading another instance of foo.o for unit testing"
[1:06] <Tv> userspace has none of those excuses
[1:07] <cmccabe> tv: so if I look at something like glibc, it will have unit tests?
[1:07] <Tv> unit testing in kernelspace is like trying to unit test ceph by dynamically loading .so's with unit tests in them to a running cluster
[1:07] <cmccabe> tv: last time I looked, there was a testsuite, but no units
[1:08] <Tv> glibc is ancient code, it's not a very good idea of best current practices
[1:08] <cmccabe> I just can't think of any major piece of C/C++ software that really has good unit test coverage
[1:09] <cmccabe> maybe Google Chrome does?
[1:09] <cmccabe> it seems like exhaustive unit testing is something people using dynamically typed languages invented to save themselves from the mistakes of their type system
[1:09] <Tv> something like samba4 might be a decent example
[1:10] <cmccabe> I guess I could be wrong, though
[1:10] <Tv> i haven't read the code, but i've followed them talk about the level of effort they put into testing
[1:10] <cmccabe> for what it's worth, I did make use of unit tests in the last piece of software I wrote in Go, which is statically typed
[1:10] <Tv> oh go has plenty of tests for itself
[1:10] <cmccabe> but not exhaustive unit tests
[1:10] <Tv> like, the standard lib etc
[1:10] <cmccabe> <--- for what I wrote
[1:11] <cmccabe> I guess I would agree that in general well-designed code tends to have unit tests
[1:11] <cmccabe> but there are some things that are difficult to unit test
[1:12] <Tv> also the other way around, unit tests help you avoid difficult-to-work-with code
[1:12] <Tv> and until you realize *that*, you haven't done it enough
[1:13] <Tv> and that's ultimately why the whole conversation about "unit tests taking developer time" is futile
[1:13] <Tv> that's like saying "designing a good architecture takes time away from typing in code"
[1:13] <cmccabe> tv: the correct number of unit tests is not infinity
[1:14] <Tv> of course not
[1:14] <cmccabe> tv: therefore, there is a tradeoff to be made between testing and doing other things
[1:14] <Tv> the correct number is related to the size of the unit under test
[1:14] <Tv> size and complexity, that is
[1:14] <cmccabe> tv: and there is also a tradeoff between unit tests and other kinds of tests
[1:14] <cmccabe> tv: which has to be made correctly for the project to really succeed
[1:14] <Tv> multiplied by a criticality and graveness of consequences factors
[1:15] <Tv> lack of both unit and integration tests causes technical debt
[1:15] <cmccabe> tv: I think it's very different for different kinds of development
[1:15] <Tv> sometimes having technical debt is the right solution
[1:15] <Tv> that doesn't mean it's not debt
[1:15] <cmccabe> tv: heh... I love that phrase
[1:15] <Tv> it won't go away, except by bancrupcy
[1:15] <cmccabe> technical debt
[1:18] <cmccabe> speaking of technical debt, s3tests with -a '!fails_on_rgw' has a bunch of failures
[1:19] <Tv> *sigh*
[1:20] <gregaf1> I'm not a big fan of their submission system, but Hadoop is pretty extensively unit tested from what I've seen
[1:21] <gregaf1> I had to write a Ceph simulator just so I could unit test our interface :x
[1:21] <gregaf1> granted that's Jave not C, but I think unit testing pervasiveness is more about the age of the project and the habits of the original team than anything else
[1:22] <cmccabe> but was the interface really complex enough to merit a unit test?
[1:22] <gregaf1> *shrug*
[1:22] <cmccabe> based on what I saw, it was literally just a pass-through to the JNI
[1:22] <gregaf1> I mean, yeah, our interface wasn't the best fit for it and it was frustrating
[1:22] <gregaf1> I'm just saying, there's a large project written in a statically-typed language
[1:23] <cmccabe> fair enough
[1:23] <gregaf1> but it did pick up a few bugs IIRC
[1:23] <cmccabe> bugs in the JNI, or in Ceph?
[1:23] <gregaf1> the interface code
[1:24] <gregaf1> and anything using external C libraries in a Java project isn't the best example of stuff that will benefit from unit tests
[1:24] <gregaf1> most things that "don't merit unit tests" don't take any time to write unit tests for, so you might as well
[1:24] <Tv> gregaf1: sadly, hadoop integration tests are really shitty; the thing falls over all the time, leaks memory, doesn't clean it's child processes, etc
[1:24] <gregaf1> lol
[1:24] <Tv> srsly one of the most often used curse words at my last job was "hadoooOOOOOP!"
[1:25] <cmccabe> that reminds me, I still need to run the integration tests for the Hadoop libceph changes
[1:25] <cmccabe> the JNI really does need unit tests, because it doesn't really benefit from strong typing
[1:26] <cmccabe> inherently it represents a bridge between two type systems
[1:26] <cmccabe> sort of floating in the void*
[1:27] <cmccabe> tv: some people seem to believe that Hypertable > Hadoop
[1:27] <Tv> nowhere near the adoption -> can't google for answers the same way
[1:28] <cmccabe> heh
[1:28] <gregaf1> Hypertable > Hadoop?
[1:28] <Tv> also gonna miss out on amazing things like Hue
[1:28] <gregaf1> that doesn't make sense to me???do you mean Hypertable > HBase?
[1:28] <Tv> i guess so
[1:28] <cmccabe> gregaf: I meant some people like Hypertable more than Hadoop
[1:28] <cmccabe> gregaf: oh, sorry, yeah
[1:29] <Tv> but don't get me started on HBase, that thing is a house of cards built on jello
[1:29] <gregaf1> lol
[1:29] <Tv> use cassandra or riak
[1:29] <gregaf1> I always assumed the only reason people use HBase is if they already have a big Hadoop cluster
[1:29] <cmccabe> tv: you mean like a bouncy castle? Those things are fun!
[1:29] <Tv> cmccabe: a bouncy castle is something you can jump around in, and it doesn't generally hurt you
[1:30] <Tv> hbase is something that manages to both move extremely slowly, and explode extremely quickly, at the same time
[1:30] <Tv> oh you've got me going, let me figure out how to draw this..
[1:31] <Tv> HBase REST API -> HBase Thrift API -> HBase SomethingServer -> HBase TabletServer -> HDFS NameNode & HDFS RegionServer
[1:32] <Tv> that's realistically how they expected you to do a simple GET of a row
[1:32] <Tv> Tablet is google terminology, i forget what hbase called it
[1:32] <cmccabe> wacky
[1:32] <cmccabe> I think Google FS also has at least three layers though
[1:33] <cmccabe> probably they leave off REST and Thrift
[1:33] <Tv> here's the equivalent for cassandra: --thrift--> Cassandra --(potentially internally rerouted to reach the node that stores the data)-->Cassandra reads from disk, writes to socket
[1:33] <cmccabe> so why doesn't everyone just use Hadoop + Cassandra?
[1:33] <cmccabe> both are Java right?
[1:33] <Tv> many people do
[1:33] <Tv> that they're java is unrelated
[1:33] <Tv> you just talk the protocol
[1:33] <cmccabe> well, if you're running a JVM already, you've paid most of the java performance tax right
[1:34] <Tv> riak is erlang+REST and it's a pleasure to use
[1:34] <Tv> cmccabe: but now you're running multiple full JVMs
[1:34] <cmccabe> I assume that HBase requires a JVM on each node doing storage
[1:34] <Tv> oh and they fight with each other on who's the king of the RAM hill
[1:34] <cmccabe> tv: oh, they can't share anything?
[1:34] <Tv> completely unrelated processes
[1:34] <cmccabe> tv: I assume intrepid sysadmins just use static memory allocation for each JVM
[1:34] <cmccabe> tv: as in set a max size
[1:35] <Tv> cmccabe: that just means optimizing them to crash early...
[1:35] <cmccabe> tv: I guess that still means page cache won't be sized intelligently
[1:35] <cmccabe> tv: I remember having to do that for eclipse
[1:35] <cmccabe> tv: set the heap size by hand
[1:35] <Tv> so it seems dbench autotest no longer completes :(
[1:36] <Tv> this is one thing we could use a non-dev QA person for
[1:37] <Tv> to run the tests often & tally the results
[1:37] <Tv> coz none of that stuff is automated yet
[1:37] <cmccabe> yep
[1:37] <cmccabe> also, even stuff that is automated requires somebody to fix the automation when it breaks down
[1:37] <Tv> that and restore sepia nodes when we wreck them
[1:38] <Tv> fixing automation = dev skills..
[1:38] <cmccabe> "automated QA" is kind of a mixed character class
[1:38] <cmccabe> not as much mana as a developer, but more HP
[1:39] <Tv> and can wield blessed weaponry of the Ops tribe
[1:39] <cmccabe> blessed or cursed?
[1:39] <Tv> one mans religion...
[1:40] <cmccabe> I think #1059 is actually causing most of the s3-tests failures for rgw
[1:41] <Tv> maybe i had more luck, yeah
[1:41] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[1:41] <Tv> i also only filed tickets on errors i could reproduce
[1:41] <Tv> and those are the ones i marked fails_on_rgw
[1:41] <Tv> maybe try constraining the number of fcgi children your apache runs to 1?
[1:41] <Tv> as a workaround
[1:42] <cmccabe> let me see
[1:42] <Tv> not sure how or anything like that
[1:42] <Tv> just saying, it's harder to race against yourself
[1:44] * monrad-51468 (~mmk@domitian.tdx.dk) Quit (Server closed connection)
[1:44] * monrad-51468 (~mmk@domitian.tdx.dk) has joined #ceph
[1:46] <cmccabe> tv: looks like "FastCgiServer /usr/bin/radosgw -processes 1" should do it
[1:52] <Tv> yup, confirmed, ceph_dbench autotest is busted even if i back out my test changes
[1:53] <cmccabe> still getting failures even with -processes 1
[1:53] <Tv> this is on cfuse, fwiw
[1:56] <Tv> http://tracker.newdream.net/issues/1063
[1:56] <Tv> no clue about why
[1:57] <cmccabe> finally actually closed a bug, 1055
[1:57] <cmccabe> although not without revealing more
[2:09] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:09] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[2:14] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[2:28] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[2:31] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[2:33] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit ()
[2:37] * MK_FG (~MK_FG@219.91-157-90.telenet.ru) has joined #ceph
[2:43] * djlee (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[2:47] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[2:49] * MK_FG (~MK_FG@219.91-157-90.telenet.ru) Quit (Read error: Connection reset by peer)
[2:50] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[3:08] * cmccabe (~cmccabe@ has left #ceph
[3:16] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[3:21] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[3:53] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[3:56] * zwu (~root@ Quit (Ping timeout: 480 seconds)
[4:21] * MK_FG (~MK_FG@ has joined #ceph
[4:53] * badari (~badari@ Quit (Server closed connection)
[4:54] * badari (~badari@ has joined #ceph
[5:00] * djlee (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:02] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) Quit (Read error: Operation timed out)
[5:14] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:19] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[5:29] * djlee (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:33] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) Quit (Read error: Operation timed out)
[5:35] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:38] * djlee2 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:39] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Read error: Operation timed out)
[5:40] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: zzZZZZzz)
[5:43] * zwu (~root@ has joined #ceph
[5:43] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[5:47] * djlee (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:52] * djlee2 (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[5:54] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[5:58] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Read error: Operation timed out)
[6:02] * yehudasa_home (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[6:02] * djlee (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[6:06] * djlee2 (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[6:07] * djlee1 (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[6:12] * djlee (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[7:39] * yehudasa_home (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[7:50] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[8:05] * lxo (~aoliva@ has joined #ceph
[8:06] * joshd (~jdurgin@adsl-75-28-69-238.dsl.irvnca.sbcglobal.net) has joined #ceph
[8:30] * lxo (~aoliva@ Quit (Read error: Operation timed out)
[8:30] * lxo (~aoliva@ has joined #ceph
[8:34] * yehudasa_home (~yehudasa_@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[8:56] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[9:04] <chraible> hi @all
[9:05] <chraible> has someone experience with using qemu-img files with ceph?
[9:05] <chraible> formats qcow2, img or raw?!
[9:10] * darktim (~andre@ticket1.nine.ch) Quit (Remote host closed the connection)
[9:22] * lxo (~aoliva@ has joined #ceph
[9:28] * votz (~votz@dhcp0020.grt.resnet.group.UPENN.EDU) Quit (Quit: Leaving)
[9:34] * joshd (~jdurgin@adsl-75-28-69-238.dsl.irvnca.sbcglobal.net) Quit (Quit: Leaving.)
[9:37] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) has joined #ceph
[9:43] <wido> chraible: You want to rung qcow2 images from the Ceph filesystem?
[9:43] <wido> why would you? Why not use Qemu-RBD
[9:44] <chraible> I have an Cloud plattform (OpenNebula) where I have to clone / copy those images from one point to ceph...
[9:44] <chraible> ceph-fs
[9:46] <chraible> OpenNebula can't work with qemu-rbd driver... it copies "hard" images from $LOCATION (location of master image) to $VMLOCATION
[9:48] * lxo (~aoliva@ Quit (Read error: Operation timed out)
[9:50] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[10:06] <wido> chraible: Ah, I get your point
[10:06] <wido> I haven't tested Qcow2 nor raw images from a Ceph filesystem. I think it should work
[10:07] <chraible> my problem is, that no one of those images located / create on my ceph-fs boot or start...
[10:07] * lxo (~aoliva@ has joined #ceph
[10:08] <wido> are there any messages in the dmesg?
[10:08] <wido> and you are sure the Ceph filesystem is working correctly? All PG's are active+clean
[10:14] <chraible> in dmesg i can't find anything
[10:14] <chraible> how can I test ceph-fs is working correctly ^^ creating an file with dd works perfect (size form 100 MB to 10 GB )
[10:15] <chraible> PG's I will look after the reboot of my servers are done :)
[10:29] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) Quit (Ping timeout: 480 seconds)
[10:34] <chraible> with ceph -w there comes following PG-statistcs " 794 pgs: 2 creating, 792 active+clean "
[10:34] <chraible> is 2 creating normal?
[10:41] <wido> chraible: I'm not sure what that means
[10:45] * Jiaju (~jjzhang@ Quit (Server closed connection)
[10:46] * Dantman (~dantman@ has joined #ceph
[10:46] * Jiaju (~jjzhang@ has joined #ceph
[10:52] * DanielFriesen (~dantman@S0106001eec4a8147.vs.shawcable.net) has joined #ceph
[10:54] * Dantman (~dantman@ Quit (Ping timeout: 480 seconds)
[11:31] * yehudasa_home (~yehudasa_@bzq-79-183-23-236.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[11:53] * yehudasa_home (~yehudasa_@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[12:16] * yehudasa_home (~yehudasa_@bzq-79-183-23-236.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[12:17] <Yulya> hm
[12:18] <Yulya> http://paste2.org/p/1400200
[12:18] <Yulya> my mds's dies with this message while doing replay
[12:19] <Yulya> *all my mds's dies :(
[12:27] <wido> Yulya: can you reproduce it?
[12:27] <wido> If so, could you start the mds with debug mds = 20 and file a issue at tracker.newdream.net?
[12:35] <Yulya> hm
[12:35] <Yulya> when i enable debug ms = 1 and debug mds = 1
[12:35] <Yulya> http://paste2.org/p/1400225
[12:36] <Yulya> looks like mds's restart after felt down
[12:37] * yehudasa_home (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[12:38] <wido> Yulya: you should check if the process is really running
[12:39] <Yulya> uh
[12:39] <Yulya> well
[12:39] <Yulya> finaly mds dies on one node
[12:39] <Yulya> http://paste2.org/p/1400230
[12:40] <wido> Yulya: I'm not an expert with the MDS, so I really have no clue
[12:40] <wido> You could report a bug, that way the devs can take a look at it
[13:31] <Yulya> omgg
[13:31] <Yulya> i couldnt even get my data back :(
[13:32] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[13:44] <trollface> Yulya:
[13:45] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[13:51] <Yulya> trollface: ?
[14:47] <Yulya> well, is there any way get my data back?
[14:48] <Yulya> maybe skip replay on this file?
[14:48] <Yulya> all mds's dies with this message http://paste2.org/p/1400324
[14:48] <trollface> there's a way to reset mds journal
[14:49] <trollface> but I am not sure it's the journal problem
[14:50] <trollface> I think you'll have to wait for gregaf1
[14:50] <Yulya> hm
[14:50] <Yulya> what I really had to lose if i reset mds journal?
[14:51] <trollface> uncommitted metadata ops
[14:54] <Yulya> well
[14:54] <Yulya> anyway in worst case i lost all data
[15:01] * Hugh (~hughmacdo@soho-94-143-249-50.sohonet.co.uk) Quit (Ping timeout: 480 seconds)
[15:09] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: zzZZZZzz)
[15:11] <trollface> Yulya: *sadface*
[15:16] * Hugh (~hughmacdo@soho-94-143-249-50.sohonet.co.uk) has joined #ceph
[15:16] * Hugh (~hughmacdo@soho-94-143-249-50.sohonet.co.uk) Quit (Remote host closed the connection)
[16:03] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[16:10] <Yulya> hm
[16:11] <Yulya> i found osd nodes wich kills mds when being started
[16:12] <Yulya> but now i cant mount ceph
[16:13] <Yulya> how can i debug cfuse?
[16:13] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[16:36] * Hugh (~hughmacdo@soho-94-143-249-50.sohonet.co.uk) has joined #ceph
[16:47] <Yulya> trollface: how can i clean mds journal?
[17:06] <trollface> let me search my logs
[17:07] <trollface> Yulya: cmds -i 0 --reset-journal 0
[17:10] <trollface> if it hangs then it's broken and you need a patch
[17:15] <Yulya> hm
[17:16] <Yulya> i couldnt google description of --reset-journal
[17:16] <Yulya> are you sure?
[17:16] <Yulya> where you get it from?
[17:17] <Yulya> oh
[17:17] <Yulya> i found, sorry
[17:27] <trollface> just grep ceph sources :)
[17:29] * Yulya is now known as Tsipa
[17:37] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:51] * greglap (~Adium@ has joined #ceph
[17:54] <Tsipa> trollface: when i'm running cmds -i 0 -c /etc/ceph/ceph.conf --reset-journal 0 i get messages like this http://paste2.org/p/1400583
[17:54] <Tsipa> how to tell him not to use auth?
[17:55] <trollface> you need to run it on the machine where you're originally running mds 0
[17:55] <trollface> or, whatever your mds is named
[17:55] <trollface> just do -i yourmdsid
[17:55] <trollface> that way it will correctly take the keyfile if needed etc
[17:56] <Tsipa> i run cmds on machine where mds.0 is declared
[17:57] <Tsipa> cmds -i 0 starts normaly
[17:57] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:59] <trollface> which version do you use? you may need greglap's flag patch
[17:59] <Tsipa> 0.27
[18:08] <sagewk> tsipa: you get auth errors with --reset-journal but without it? (are all other args the same?)
[18:15] <Tsipa> sagewk: yeah
[18:15] <Tsipa> sagewk: cmds -i 0 works fine, but with --reset-journal i got auth errors
[18:37] <sagewk> Tsipa: weird. we'll try to reproduce it here.
[18:38] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:42] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Quit: Ex-Chat)
[18:42] <Tsipa> sagewk: trollface already send me patch to src/mds/Resetter.cc
[18:42] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[18:42] <sagewk> oh nice, did it work?
[18:43] <Tsipa> i'l try it in 20 minutes
[18:44] <Tv> gitbuilder filled its disk, fixing..
[18:51] <Tsipa> sagewk: yeah, it works
[18:51] <sagewk> cool. share the patch?
[18:53] <trollface> Tsipa: did it fix the problem? :)
[18:53] <trollface> sagewk:
[18:53] <trollface> - messenger->start(true, getpid());
[18:53] <trollface> + messenger->start(false, getpid());
[18:53] <trollface> in Resetter
[18:54] <sagewk> oh right. thanks
[18:54] <sagewk> actaully looks like that's already fixed in the master branch
[18:55] <Tsipa> trollface: yeah!
[18:55] <Tsipa> trollface: tnx you very much
[18:55] <trollface> sagewk: but not in stable
[18:55] <trollface> I'm tracking stable so I had to reapply it
[18:56] <trollface> anyway
[18:56] <trollface> sagewk: will you be around 2 hours from now?
[18:56] <sagewk> yeah
[18:56] <trollface> I was having some fun with cls_rbd
[18:56] <trollface> so I was hoping to debug it properly as yesterday I was distracted by something else
[18:56] <trollface> okay then
[18:56] * trollface off to the meeting
[18:58] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:02] * gregaf1 (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[19:03] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has joined #ceph
[19:05] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:06] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:22] <sagewk> 10:30!
[19:23] * lidongyang (~lidongyan@ Quit (Server closed connection)
[19:23] * lidongyang (~lidongyan@ has joined #ceph
[19:47] * yehudasa_hm (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[19:48] * yehudasa_hm (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) Quit (Read error: Connection reset by peer)
[19:48] * yehudasa_home (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) Quit (Read error: Connection reset by peer)
[19:53] * aliguori (~anthony@ has joined #ceph
[19:54] * yehudasa_home (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[20:12] <Tv> huh an autotest just gave me degraded PGs on a brand new cluster with 1 osd
[20:13] <Tv> 11:12:28 DEBUG| Ceph health: HEALTH_WARN 18 pgs degraded, 21/42 degraded (50.000%)
[20:13] <joshd> tv: that's expected with a replication level of 2
[20:13] <gregaf> yeah, that's supposed to
[20:13] <Tv> ohh
[20:13] <Tv> did that change recently?
[20:13] <Tv> like, dd ceph health behavior change
[20:13] <gregaf> nope, been that way forever
[20:13] <gregaf> oh, yeah, I think ceph health did change
[20:13] <Tv> because i recall being able spin up a 1-osd cluster on sepia
[20:14] <Tv> yeah
[20:14] <Tv> now you can't
[20:14] <Tv> oh well, restarting the test..
[20:14] <gregaf> I don't think it reported pg degradation at all and now it does
[20:17] <Tv> yeah i remember that being mentioned.. explains the symptoms perfectly
[20:18] <Tv> sagewk, *: Making sure you see this: http://cloudfs.org/2011/05/red-hat-summit-slides/ -- slide 10 is a good quick description
[20:19] <sagewk> tv: k
[20:20] <Tv> let's just say that i won't be tracking it as much as i used to
[20:20] <Tv> (the author has a nice blog, but the software looks uninteresting)
[20:28] <cmccabe> tv: so cloudFS is some kind of middleware layer over glusterFS ?
[20:28] <Tv> so it seems
[20:28] <Tv> sounds a lot like the FiST stackable filesystems stuff
[20:28] <cmccabe> tv: he seems to use the project names almost interchangably
[20:29] <Tv> except not in kernel maybe?
[20:29] <joshd> tv: glusterfs is all userspace
[20:29] <cmccabe> tv: I don't think any of gluster is in the kernel
[20:30] <bchrisman> make testceph now outputs a libtool script into a file named testceph? (configure ???with-debug)
[20:30] <Tv> also sounds a lot like RAIF
[20:30] <Tv> bchrisman: libtool does that.. run the script and it'll run the real thing underneath
[20:30] <cmccabe> tv: looks like cloudFS is a thin encryption shim above gluster
[20:31] <Tv> bleh EIO
[20:31] <bchrisman> Tv: I see that??? generally I package stuff up into an rpm and deploy to a cluster??? this needs to have source directory access on cluster then?
[20:31] <Tv> i seem to be having difficulty keeping a cluster healthy, lately
[20:31] <Tv> bchrisman: not if you make install/rpm/deb it
[20:32] <Tv> bchrisman: basically the shell scripts set up things like LD_LIBRARY_PATH so the executables will work even in the source tree
[20:33] <Tv> bchrisman: libtool/automake are layered design, much like an onion -- as in, it'll make you cry
[20:33] <bchrisman> heh
[20:38] <bchrisman> there should be a line for that in ceph.spec.in then? at least.. for debug?
[20:39] <bchrisman> hmm.. I should go check my rpm first.. yeah
[20:42] <cmccabe> tv: I modified test-obsync.sh to use the same configuration file format as s3-tests
[20:43] <cmccabe> tv: hopefully this will make it easy for people other than me to run it
[20:43] <cmccabe> tv: er, test-obsync.py
[20:58] <Tv> nice
[21:04] * tjfontaine (tjfontaine@tjfontaine.chair.oftc.net) has joined #ceph
[21:04] <tjfontaine> anyone know for whom is a NAT gateway for?
[21:05] <tjfontaine> there are a few users in this channel from that endpoint
[21:06] <tjfontaine> badari and aliguori for example
[21:06] <aliguori> ibm
[21:07] <aliguori> i would guess
[21:07] <tjfontaine> ok, thanks, they should update the whois information and/or rdns :)
[21:07] <aliguori> heh, that's not a terribly easy thing to do at a place like ibm ;-)
[21:08] <tjfontaine> yes well, I would execpt whois to at least to be mapped :P
[21:12] <tjfontaine> aliguori: ok well you or any other cohorts shouldn't have an issue connecting in the future, unless you're at some conference or so
[21:12] <tjfontaine> I'm also slightly concerned that this channel is not registered with services
[21:14] <trollface> so, currently, if I use btrfs I still need journal
[21:16] <bchrisman> running testceph, seeing Operation not permitted in mount call: http://pastebin.com/R75a9iy4
[21:16] <bchrisman> wondering if anybody who's run it before knows what I might be hitting ehre?
[21:17] <Tv> bchrisman: is your server requiring crypto auth
[21:17] <Tv> bchrisman: perhaps you need to make it read your ceph.conf
[21:17] <bchrisman> yeah...
[21:17] <bchrisman> [global]
[21:17] <bchrisman> auth supported = cephx
[21:17] <bchrisman> debug ms = 0
[21:17] <bchrisman> keyring = /etc/ceph/keyring.bin
[21:18] <bchrisman> hmm.. I can trace it and see if it's opening the file properly
[21:18] <Tv> well i see the same thing
[21:19] <cmccabe> bchrisman: you can use cconf to see what configuration values you are getting
[21:20] <Tv> cmccabe: vstart and ./testceph fails too
[21:20] <cmccabe> tv: on head of line?
[21:20] <Tv> e97ce8ec87aada43c8e27ccd05496f4614980837 from yesterday i think
[21:20] <Tv> grabbing & compiling
[21:25] <trollface> any ideas how to debug this:
[21:25] <trollface> 2011-05-05 23:21:23.642451 7ff72c449740 librbd: failed to assign a block name for image
[21:25] <trollface> create error: Input/output error
[21:25] <trollface> ?
[21:26] <trollface> sagewk:
[21:29] <cmccabe> trollface: he's probably at lunch at the moment
[21:29] <sagewk> back now
[21:29] <sagewk> it probably means the rbd class isn't loaded?
[21:29] <cmccabe> trollface: joshd might also have some insight into librbd
[21:29] <Tv> cmccabe: testceph fails on latest master
[21:30] <cmccabe> tv: is this part of the everything failing problem with auth?
[21:30] <Tv> cmccabe: ceph -s works, auth is fine
[21:30] <sagewk> bchrisman,cmccabe: btw i was getting auth errors in testceph the other day too, didn't track it all the way down tho
[21:30] <joshd> trollface: sage is right, that happens when it's not loaded or not activated -- see http://ceph.newdream.net/wiki/Rbd#Loading_RBD
[21:30] <sagewk> (ceph with the same name and key would work fine.. it was weird)
[21:30] <Tv> cmccabe: perhaps testceph uses wrong id or something?
[21:31] <bchrisman> cool??? I'm not smoking something then.. good news that.
[21:31] <trollface> 2011-05-05 23:30:46.749050 mon <- [class,list]
[21:31] <trollface> 2011-05-05 23:30:53.999851 mon2 -> 'installed classes:
[21:31] <trollface> rbd (v1.3 [x86-64]) [active]
[21:31] <trollface> ' (0)
[21:31] <cmccabe> my unit tests for obsync are running into some strange difficulties
[21:31] <cmccabe> it seems that you can "give away" buckets
[21:31] <cmccabe> and the test somehow gives away a bucket to someone and then can't get it back
[21:31] <Tv> cmccabe: on aws?
[21:32] <cmccabe> it's bizarre
[21:32] <cmccabe> like the ndncmc1 bucket used to be mine, but now I get 403 when trying to list it
[21:32] <cmccabe> and it doesn't show in from listbuckets
[21:35] <Tv> cmccabe: can you check whether vstart and testceph works for you or not?
[21:35] <cmccabe> tv: the first time it happened, I was surprised that someone else had picked the bucket name "ndn-cmccabe1"
[21:35] <cmccabe> tv: then I started to get suspicious :)
[21:35] <Tv> cmccabe: i wonder if forbidding acls could make it not show up in listbuckets, that sounds wrong
[21:36] <Tv> cmccabe: but getting 403 from actual bucket operations sounds like what it should do if you forbid it in acls accidentally
[21:36] <cmccabe> tv: vstart works for me, and testrbd works
[21:36] <Tv> cmccabe: testceph
[21:36] <cmccabe> tv: testceph works as well
[21:36] <Tv> cmccabe: huh.. are you passing any args?
[21:36] <cmccabe> no
[21:36] <Tv> cmccabe: is your ceph.conf the standard thing?
[21:37] <cmccabe> it was generated by vstart?
[21:37] <Tv> cmccabe: any args to vstart?
[21:37] <cmccabe> -d -n
[21:37] <bchrisman> you guys seeing the same thin in that ceph_mount is erroring?
[21:37] <Tv> cmccabe: try -x
[21:37] <sagewk> mine fails (with -x)
[21:37] <Tv> bchrisman: yeah i think it's the testceph not doing auth
[21:38] <Tv> bchrisman: quick workaround, turn off cephx
[21:38] <cmccabe> testceph fails when cephx = 1 apparently
[21:38] <cmccabe> for me
[21:38] <Tv> yeah that's what we're seeing
[21:38] <Tv> bchrisman, cmccabe: can you guys work out a ticket for it etc?
[21:39] <cmccabe> tv: it looks like a crypto problem
[21:39] <bchrisman> does ceph work without authx now? I remember before it was required..
[21:39] <cmccabe> tv: but if you want, I can do it
[21:39] <bchrisman> yeah.. I'll create a ticket
[21:39] <Tv> bchrisman: works just fine, test automation uses cephx
[21:39] <Tv> cmccabe: the auth code is fine, testceph or libceph is not doing something right
[21:39] <sagewk> it's not crypto per se.. teh same keyring and name authenticates fine for rados, ceph. somewhere the right info is not getting passed into libceph's MonClient instance
[21:40] <cmccabe> frankly I am annoyed by the way crypto interacts with the common code
[21:40] <cmccabe> stuff like g_keyring shouldn't be global
[21:40] <sagewk> yeah, i don't think that's the issue here tho
[21:40] <cmccabe> and ceph::crypto::init looks awfully like a singleton, but isn't actually guarded by a mutex
[21:41] <Tv> it's guarded by the calling code, currently
[21:41] <cmccabe> that is wrong
[21:41] <Tv> but that is not this bug
[21:41] <cmccabe> anyway, I'll see if I can fix
[21:41] <Tv> i don't see much benefit in adding the locks until the time the wrapping locks are taken out
[21:45] <sagewk> libceph is getting a secret of AAAAAAAAAAAA
[21:46] <sagewk> is that a magic value?
[21:46] <sagewk> oh
[21:47] <sagewk> keyring_init is calle din ceph_create.. shouldn't that be in ceph_mount, after the config is read, args parsed, etc.?
[21:47] <cmccabe> sagewk: sounds right
[21:47] <cmccabe> sagewk: in general keyring_init is doing a lot of gross things with g_conf and g_keyring
[21:47] <sagewk> yeah that fixed it for me
[21:48] <sagewk> i'll push a fix
[21:49] <cmccabe> librados already has the keyring_init in connect, so that's not a problem
[21:50] <sagewk> bchrisman: should be good to go now
[21:50] <bchrisman> sagewk: cool.. we'll get that on next build.
[21:52] <sagewk> tv: was that 606 seconds of cleanup (i.e. 1206) or finish at time 606 (6 seconds of cleanup)?
[21:52] <Tv> sagewk: good question, re-checking the logs
[21:52] <sagewk> ah, yeah..6 seconds of cleanup
[21:52] <sagewk> that's reasonably enough
[21:52] <Tv> sagewk: yeah it's total, completely fine
[21:53] <sagewk> phew :) you keep freaking me out!
[21:53] <Tv> sagewk: the reason i aborted the cfuse run was really my ls hanging
[21:53] <Tv> and it had 744-600 = 144 seconds of cleanup
[21:53] <Tv> so it probably was really boken
[21:53] <Tv> i always make a tpyo when i try to spell borken
[21:58] <gregaf> Tv: remind me where in there are supposed to be the logs?
[21:58] <Tv> hmmmh a mount/umount loop fails now too: http://autotest.ceph.newdream.net/afe/#tab_id=view_job&object_id=559
[21:58] <Tv> gregaf: it seems the autotest log gathering broke, i'm debugging
[21:58] <Tv> gregaf: right now all i can say is that it works some of the time
[21:59] <gregaf> oh, I figured it was working since you referred us to it again :)
[21:59] <Tv> well there's different kinds of logs
[21:59] <Tv> the stdout/stderr capture works
[21:59] <Tv> the write to log file part seems to be misbehaving
[22:03] <Tv> ah might be related to user clicking abort
[22:03] <Tv> as in, any job that is actually aborted, gets no logs
[22:03] <Tv> aka the code is plugged into the wrong hook :(
[22:04] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[22:09] <cmccabe> tv: I think I figured out what was happening with this test script
[22:09] <cmccabe> tv: I was just creating the other bucket as a different user
[22:09] <cmccabe> tv: no ACL problem, phew
[22:09] <Tv> <3 occam's razor
[22:10] <Tv> $ git grep 'def abort('|wc -l
[22:10] <Tv> 8
[22:10] <Tv> oh autotest
[22:11] <Tv> i'm convinced there's a minotaur in the source, somewhere
[22:12] <sagewk> cmccabe: oh good. :)
[22:12] <sagewk> cmccabe: thats 1059 right? i can close it out?
[22:18] <cmccabe> sagewk: not at all
[22:18] <cmccabe> sagewk: I'm just debugging obsync now
[22:18] <cmccabe> sagewk: rgw is not involved
[22:19] <sagewk> oh well. :(
[22:20] * yehudasa_home (yehudasa_h@bzq-79-183-23-236.red.bezeqint.net) Quit ()
[22:20] <cmccabe> nice try :)
[22:20] <cmccabe> really, just running s3-tests is enough to see what's going on with 1059
[22:22] * yehuda_hm (~yehuda@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[22:24] <yehuda_hm> cmccabe: are you running s3-tests against objects?
[22:24] <cmccabe> rgw-cmccabe
[22:29] <yehuda_hm> cmccabe: what backend does it use?
[22:29] <cmccabe> a local rados cluster
[22:29] <yehuda_hm> how many osds?
[22:30] <cmccabe> one
[22:30] <yehuda_hm> what's the ip address?
[22:31] <cmccabe> https://uebernet.dreamhost.com/index.cgi?tree=dev.paste&action=view&id=3024
[22:31] <cmccabe>
[22:31] <cmccabe> if you change anything in src/ceph, be aware that I'll probably blow it away on the next rsync
[22:33] <yehuda_hm> it's not accessible, is the vm running?
[22:33] <cmccabe> er, sorry
[22:33] <cmccabe>
[22:33] <cmccabe> looks like my paste didn't quite make it
[22:34] <yehuda_hm> and where are you running the s3-tests?
[22:35] <cmccabe> metropolis
[22:35] <Tv> gregaf: current understanding: ceph logs are lost for the jobs that aborted. (even when you click Abort in web ui, some nodes might have completed the job already; those will have logs collected)
[22:35] <Tv> gregaf: fixing that now, somehow... :-/
[22:39] <Tv> cmccabe: s pthread_mutex_t ok without any initialization?
[22:39] <Tv> *is
[22:39] <cmccabe> sorry, should have PTHREAD_MUTEX_INITIALIZER
[22:40] <cmccabe> added
[22:43] <yehuda_hm> cmccabe: looking at the bug report, it looks to me that all the issues are on the bucket creation, is that true?
[22:44] <cmccabe> yehuda_hm: I haven't looked at all those tests closely
[22:44] <yehuda_hm> yeah, it fails when creating the bucket.. not really when writing objects
[22:45] <yehuda_hm> which we'd made the bucket creation async
[22:45] <cmccabe> yehudasa: looks like all the tests that failed on my latest run involve bucket creation
[22:45] <yehuda_hm> which <= as
[22:45] <cmccabe> yehudasa: however, I've also seen things like creating and object and then setting an ACL on that object fail
[22:46] <cmccabe> yehudasa: which makes me think there are bigger problems with however we're enforcing read-after-write consistency
[22:46] <yehuda_hm> are you sure that was the case there?
[22:46] <cmccabe> also keep in mind I'm running with max processes = 1 because I thought that would help
[22:46] <cmccabe> yehudasa: I haven't looked at those specific test failures
[22:47] <cmccabe> yehudasa: except to note that they exist, and there's little point trying to debug both obsync and rgw simultaneously
[22:47] <yehuda_hm> all the backtraces in the tracker are for bucket creation
[22:48] <cmccabe> http://tracker.newdream.net/attachments/207/cmccabe-rgw-tests-2011-05-05.txt
[22:48] <cmccabe> looks like we got UnresolvableGrantByEmailAddress, NoSuchKey, NoSuchBucket, etc
[22:49] <cmccabe> I dunno, maybe that is all somehow bucket-realted
[22:49] <cmccabe> related
[22:49] <yehuda_hm> the NoSuchKey, NoSuchBucket are bucket related
[22:50] <cmccabe> yehuda_hm: s3tests conf is here: https://uebernet.dreamhost.com/index.cgi?tree=dev.paste&action=view&id=3025
[22:51] <yehuda_hm> probably the UnresolvableGrant.. is just a bad error return code, but probably also related to async bucket creation because we set acl on the bucket
[22:51] <cmccabe> sorry, this one is better
[22:51] <cmccabe> https://uebernet.dreamhost.com/index.cgi?tree=dev.paste&action=view&id=3026
[22:56] <yehuda_hm> yeah.. this one works
[22:57] <Tv> cmccabe: it seems your crypto mutex change broke the nss fork kludge; make check fails now
[22:58] <Tv> cmccabe: because it inherits the crypto_init flag over fork, and shutdown doesn't clear it; i can fix that
[23:01] <yehuda_hm> cmccabe: the issue for the UnresolvableGrantByEmailAddress is that user doesn't have email address configured
[23:02] <yehuda_hm> cmccabe: radosgw_admin user modify --uid=bar --emai=bar@example.com will fix that one
[23:02] <yehuda_hm> the rest are bucket creation related, and we should think whether we want to solve it (make it sync again) or not.
[23:03] <sagewk> yehuda_hm: we realized bucket creation is slow on amazon too, right?
[23:03] <yehuda_hm> sagewk: amazon states that bucket creation is not scalable and should never be in a critical path
[23:04] <sagewk> yehuda_hm: which means we should probably just make it sync. altho really that's a bit tricky, because we need to probe every pg to make sure all the osds have the latest map. doing an 'ls' will do the trick
[23:05] <cmccabe> tv: so shutdown needs to clear the flag then?
[23:05] <sagewk> cmccabe: hmm, for the backup use case: obsync might be a bit awkward there, since you need to have the proper rgw credential for each bucket, right?
[23:05] <cmccabe> tv: do you want me to fix it or are you going to
[23:05] <Tv> cmccabe: yeah, done & tested already
[23:05] <gregaf> sagewk: I thought the problem with sync pool creates was it timed out?
[23:05] <cmccabe> tv: k
[23:06] <cmccabe> sagewk: yes, you can't write to a bucket without credentials
[23:06] <cmccabe> sagewk: or read from a bucket
[23:07] <cmccabe> sagewk: maybe the awkwardness could be reduced by having a configuration file that mapped user_id to skey, akey
[23:07] <cmccabe> sagewk: that way we could at least look at who owned the bucket, and try to be that user
[23:07] <cmccabe> sagewk: rather than relying on the sysadmin to keep track of that
[23:08] <sagewk> cmccabe: alternatively we can add basic incremental functionality to rados import/export
[23:08] <cmccabe> sagewk: ?
[23:08] <cmccabe> sagewk: all obsync backups are incremental
[23:08] <sagewk> for the bucket creation: maybe having rgw check with the mon for a new map on bucket dne is cheaper/faster.
[23:08] <cmccabe> sagewk: the functionality that's lacking from rados import/export is ACLs
[23:09] <sagewk> the rados tool import/export i mean (which can use a 'root' or 'read anything' credential and blindly back up everything in sight)
[23:09] <sagewk> it can import/export the raw xattrs, though, and capture that implicitly
[23:10] <cmccabe> sagewk: there is no such thing as a raw xattr
[23:10] <cmccabe> sagewk: or at least not a usable raw xattr
[23:10] <yehuda_hm> sagewk: I don't think you want to do that on every bucket that wasn't found
[23:10] <cmccabe> sagewk: are you talking about obsync's ACL support for rados again?
[23:10] <cmccabe> sagewk: that is done, just haven't had time to debug it
[23:10] <sagewk> the rados object attrs. this would be beneath the rgw layer entirely, backing up/restoring the rados pool directly.
[23:11] <gregaf> yehuda_hm: it's gotta be cheaper than forcing every OSD to get an up-to-date map on every bucket create though, right?
[23:11] <cmccabe> sagewk: rados ACL translation is already done in wip-obsync-rados2
[23:11] <sagewk> gregaf: it's only the osds that have pgs for the new bucket (8 by default), so it could be worse, i guess
[23:11] <cmccabe> sagewk: again, I just need time to debug it
[23:11] <gregaf> and hell, even forcing the new osdmap out to all the OSDs doesn't guarantee each gateway gets it ??? what if there's no activity on the gateway between when the bucket is created and when the access request comes in?
[23:11] * darkfader (~floh@ Quit (Server closed connection)
[23:11] <cmccabe> sagewk: is what your proposing an alternative to wip-obsync-rados2?
[23:12] * darkfader (~floh@ has joined #ceph
[23:12] <cmccabe> *you're
[23:12] <sagewk> oh yeah, gregaf is right, the bucket dne comes from the client osdmap, osds don't even matter.
[23:13] <gregaf> so unless we want to have inter-rgw communications going on they HAVE to ask the monitor if we're going for strict read-after-write consistency in a multi-rgw scenario
[23:13] * tjfontaine (tjfontaine@tjfontaine.chair.oftc.net) has left #ceph
[23:13] <sagewk> cmccabe: the rados acl translation wouldn't be needed if we back up the raw rados pool directly. it would have to be restored directly to a rados pool (bc there's no awareness of acls etc) but for backups that's ok.
[23:13] <yehuda_hm> yeah.. though I'm still not sure whether it'd be a good idea. I'm more concerned about malicious clients
[23:14] <cmccabe> sagewk: we could write a special-purpose tool that just uses librados to copy an entire rados pool
[23:14] <gregaf> yehuda_hm: yeah, that concerns me too but I'm not sure there's any way around it with the semantics we're talking about
[23:14] <sagewk> that's 'rados import' and 'rados export'?
[23:15] <cmccabe> sagewk: so we already this then
[23:15] <cmccabe> *have this
[23:15] <sagewk> i think the monclient protocol only has a single request in flight (per map type), so that limits the load on the monitor, at least
[23:15] <gregaf> and I don't think that giving malicious clients the ability to spam the monitor with trivial messages is worse than what they can already do anyway?
[23:16] <sagewk> cmccabe: yeah, except it doesn't do the incremental part.
[23:16] <yehuda_hm> sagewk, gregaf: we can check whether the bucket acl object exist before requesting a new map
[23:16] <yehuda_hm> and write that object synchronously
[23:16] <gregaf> write it?
[23:16] <sagewk> yehuda_hm: oh, good idea
[23:16] <yehuda_hm> that way we avoid going to the monitor
[23:17] <yehuda_hm> i mean.. avoid going to the mon *always*
[23:17] <gregaf> is redirecting from the monitor to the (same!) OSD worth anything to us?
[23:17] <yehuda_hm> gregaf: different scaling properties
[23:18] <cmccabe> sagewk: I don't think rados export preserves any metadata at all
[23:18] <sagewk> cmccabe: so basically, using obsync: pros=backup can sync to anything, not just rados, cons=need to figure out credentials for each bucket. radostool: pros=easy and simple, cons=need to implement incremental support, lfn support.
[23:18] <sagewk> and xattrs :)
[23:18] <gregaf> yehudasa_hm: I understand that, but what I'm saying is we have 3 monitors and we'll probably have all the ACLs on 3 OSDs too...
[23:18] <Tv> yehuda_hm: it seems your choice is to either suffer at every access to bucket that doesn't seem to exist (need to double-check), or to suffer at bucket creation time; of those, bucket creation costs attacker money...
[23:18] <gregaf> and by going to the OSDs we're more likely to need to hit disk if the full dir contents aren't in cache
[23:18] <yehuda_hm> gregaf: we have an object per bucket\
[23:18] <cmccabe> sagewk: LFN support needs to be added to obsync as well
[23:19] <gregaf> yehuda_hm: yes, and those objects are stored in a teeny little default-sized pool, right?
[23:19] <Tv> cmccabe: even more reason for me to make noises about librgw...
[23:19] <cmccabe> tv: I mean LFN in the sense of local files
[23:20] <Tv> cmccabe: ah yeah that's different
[23:20] <cmccabe> tv: LFN was implemented in RADOS itself, but that is different
[23:20] <gregaf> I mean if we want to make a habit out of not going to the monitors that's fine with me, it's just this is a roundabout way of checking existence that I don't think is going to buy us anything in the practical sense
[23:20] <sagewk> sigh.. how wrong i was when i thought FileStore would be the only thing that needs it
[23:20] <gregaf> we could just not support long file names for a local target of obsync...
[23:21] <gregaf> that's not a terribly important use case for us, is it?
[23:21] <gregaf> or are we planning to do backups to local disk with it?
[23:21] <cmccabe> maybe we should only support DOS 8.3 names
[23:21] <yehuda_hm> gregaf: ahmm..right
[23:21] <cmccabe> it should be enough for anyone
[23:21] <cmccabe> also objects larger than 64k are probably excessive
[23:22] <cmccabe> "64k should be enough for anyone"
[23:22] <Tv> fix one thing, re-break another :(
[23:22] <Tv> now that i disabled gzipping, the log files are not readable again
[23:23] <gregaf> cmccabe: we wrote obsync to fill a business need and I'm not sure that the local filesystem backend fills any business need for us (as opposed to testing needs), is all I'm saying
[23:23] <sagewk> for now lfn support in obsync seems like a low priority. unless it's for backups (the only time we need the local target). so it all depends on whether we want rados import/export or obsync for this
[23:23] <cmccabe> sagewk: doesn't librados also need to authenticate?
[23:24] <sagewk> any opinion on which is better for the backup use case?
[23:24] <cmccabe> sagewk: it seems like that would have all the same auth issues as obsync
[23:24] <sagewk> cmccabe: yeah but it can use a single crediential that lets it read anything
[23:24] <cmccabe> sagewk: need to remember the keys, etc
[23:24] <cmccabe> sagewk: another thing is that the rgw <-> rados translation layer may change over time
[23:24] <sagewk> oh, i guess we can do the same with the obsync rados target
[23:25] <cmccabe> sagewk: that is an issue for either obsync or "rgw export"
[23:25] <cmccabe> sagewk: one nice thing about a C++ program is that crashes in librados would be easier to debug
[23:25] <sagewk> but not for 'rados export' (unless you bacup before a flag day and restore after)
[23:26] <cmccabe> sagewk: so you're just talking about really an enhanced rados export
[23:26] <cmccabe> sagewk: with no rgw-specific functionality
[23:26] <cmccabe> sagewk: that forces the admin to know about things like the .rgw bucket?
[23:27] <cmccabe> sagewk: I guess maybe if you create a bucket using rgw, then you can get away with using "rados import" to fill it?
[23:27] <sagewk> not even that.. it just imports/exports a raw pool. it just needs incremental and attr support.
[23:27] <cmccabe> sagewk: I think the answer is that you can... but yehuda would have to answer that question
[23:27] <cmccabe> sagewk: I can spend an hour or two on it, and see if it's easy
[23:27] <cmccabe> sagewk: rados export I mean
[23:27] <sagewk> i'll make a bug for it
[23:28] <cmccabe> sagewk: it sounds pretty easy, and does avoid the thorny ACL translation issues
[23:28] <cmccabe> sagewk: but obsync will still be necessary for moving data into the system from s3, etc.
[23:28] <sagewk> yeah, and is a useful tool in general
[23:28] <sagewk> right
[23:31] <yehuda_hm> gregaf, sagewk: pg can be reconfigured, however, maybe it'd be a good idea to not create a single bucket for access-keys and buckets but multiple
[23:31] <sagewk> we can make that bucket have a high replica count, and allow reads from replicas
[23:31] <yehuda_hm> having the bucket name some suffix, e.g., the first two letters of the bucket name
[23:31] <sagewk> or put it on special fast osds or something
[23:32] <sagewk> if we can't scale a single pool we have failed elsewhere :)
[23:33] <gregaf> hehe, yes
[23:33] <gregaf> I just wonder if the system is sufficiently attacker-proof in other areas that switching from monitor to OSDs matters ;)
[23:36] * yehuda_hm (~yehuda@bzq-79-183-23-236.red.bezeqint.net) Quit (Quit: Leaving)
[23:36] * yehuda_hm (~yehuda@bzq-79-183-23-236.red.bezeqint.net) has joined #ceph
[23:39] <gregaf> Tv: do you know roughly how long those dbench runs took before they failed?
[23:39] <Tv> gregaf: 744 seconds was where i aborted the first run
[23:40] <Tv> that's in the log file
[23:40] <Tv> http://autotest.ceph.newdream.net/afe/#tab_id=view_job&object_id=560 is still running
[23:40] <gregaf> really?
[23:40] <Tv> http://autotest.ceph.newdream.net/results/560-tv/group0/sepia63.ceph.dreamhost.com/debug/client.0.log
[23:40] <gregaf> I've got one that was in "warmup" for longer than that
[23:40] <Tv> 1.5 hours in cleanup for job 560 ;)
[23:41] <gregaf> dear god that was slow
[23:42] <Tv> gregaf: perhaps dbench never leaves warmup until it reaches a certain level of MB/s?
[23:42] <gregaf> well my local run is going a lot faster than that autotest run did
[23:43] <gregaf> it's going at values of entire Megabytes per second!
[23:44] * Meths_ (rift@ has joined #ceph
[23:50] * Meths (rift@ Quit (Ping timeout: 480 seconds)
[23:51] <yehuda_hm> Tv, gregaf: probably one of the dbench operations hung.. dbench us not assuming anything about MB/s
[23:51] <gregaf> yeah, I didn't realize we had an op counter but we do and it's stuck at 2
[23:52] <yehuda_hm> yeah.. probably failed to create the directory
[23:52] <gregaf> it works fine when I run it locally, I'm trying to dig up the autotest config for it
[23:52] <Tv> heh
[23:52] <yehuda_hm> you can look at the clients.txt file to see what it tried to do
[23:52] <yehuda_hm> but the fs is probably defunct?
[23:52] <gregaf> yeah, where's that file?
[23:52] <Tv> gregaf: btw can you point me to how to read the counter
[23:52] <yehuda_hm> how did you install dbench?\
[23:53] <gregaf> Sage tells me it's just an operation counter
[23:53] <gregaf> yehuda_hm: well for me I just apt-get installed it
[23:53] <gregaf> but autotest has its own clients.txt which I assume is different
[23:53] <Tv> for autotest, there's a tarball in client/tests/ of autotest.git
[23:53] <yehuda_hm> gregaf: try /usr/share/dbench
[23:54] <gregaf> Tv: do you have that git url handy?
[23:54] <Tv> https://github.com/tv42/autotest/tree/ceph/client/tests/dbench
[23:55] <gregaf> thanks
[23:55] <sagewk> any bets on how painful building on opensolaris will be?
[23:56] <gregaf> ???building what, exactly?
[23:57] <gregaf> hmm, they seem to be the exact same client.txt
[23:57] * aliguori (~anthony@ Quit (Remote host closed the connection)
[23:57] <gregaf> unless the one that autotest is using is one that isn't in the tarball
[23:58] <Tv> sagewk: i'll bet the sanity of the developer working on that..
[23:58] <sagewk> we'll soon see, kitchen is trying it now
[23:58] <gregaf> are we talking about cfuse, kernel client, OSD???.?

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.