Show newer

@pganssle

Actually I came up with a rather simple solution early on that does work... it costs me a little in performance but nothing too major and keeps the code elegance I would like.. Im mostly just grumpy that it was needed and that i had to take that performance hit, minor as it is.

My solution was to just write an array caching class that wraps any array-like object and cashes the results so calling the same index a second time doesnt hit the underlying object. I needed a sense of caching anyway even without multiprocessing because the whole architecture is basically a stack of wrapped array-like objects each level doing additional operations (for example one layer when wrapped around an array provides a "view" of the array as its moving average, without itself containing any internal variables, it calculates on the fly and pass through). So even without multi processing there was a need to write a type of wrapper that caches at specific layers because the layer above it might re-read several values many times and dont want to re-calculate every time.

All I did was make it so these caches have a drop-in equivalent that uses shared memory arrays rather than plain arrays when in multithreaded mode and now the whole object can pass across multithreading boundaries without needing any special nonsense. It works well enough for now.

@nate_river

@Zest

I was in exactly your shoes for many years, almost exactly. I had all the basic ideas but they never fit together just right and no one who knew enough came along (until far too late) to help me put the pieces together. I'm glad I could do that for you.

@Science

@pganssle I didnt realize you were an actual python dev... in that case I understand why your earlier response to my statements were taken personally and seen as rude. I apologize for that. As I said i should have kept it to the technicals.

@fluiddyn@pythondevs.social

yes moving to a non-GIL locked python implementation of JITC is being considered. But you have to remember my **entire** code base is pretty much one big cpu limited algo that runs in real time... so if i really had to export the CPU intensive parts outside of python proper then very little of what is left would even be python... Good advice for most apps I am sure, but in this case if that is really the end result then I'd likely be better off just coding in a different language.

> In any case try to share a simple code which has the issues that you talk about, next time!

Since this is algorithmic trading i sadly can not share the code as the algo is top secret due to how successful it has been for me. thankful I am not hitting any issues I cant solve. I have so far been able to see how to use python to solve most of my issues and in doing so have managed to continually improve performance.

My complains are not so much about how/if I can achieve the result I want, its more so that I just do not like the way python makes me do it to get there.

@pganssle

@pganssle Not really forced to use python, no.. there was some pressure to use it simply because there were more python libraries for algorithmic trading (specifically to access the broker APIs themselveS) than other languages.. but if i was really against python that would not have stopped me, especially considering that most of the project is hand rolled anyway (algorithms of my own design).

> I think you may have a better experience if instead of complaining about why it doesn't fit your preconceptions you asked, "How do people do X?" Usually if other people complain, "Oh yeah that's basically impossible", then yeah it's a sore spot

That would depends on the intended goal of complaining... If you are talking about how people might respond to me, then yes, I agree. If i were in a IRC room asking for python help, for example, my tone would be very different and i wouldnt complain so directly. On here however my goal isnt really to get help (though advice and help is always welcome and considered).. its mostly me complaining because it makes me laugh, starts some decent conversations,and it feels good to vent to my friends.

Now if what you mean is that my own mentality might be different if i didnt complain and i might be more productive.. well I'd agree with you if we were talking about an all encompassing mentality. but generally thats not how it tends to play out with me... sure I complain but in the same breath im googling how to do shit in python the right way, looked at 20 different examples, and pulling those that appear most elegant. I've rewritten this code probably 5 times in various ways as I do that. So trust me, while complaining might be off putting to you and others who might be more able or willing to help (and that is an unintended side effect), it isnt really slowing me down when it comes to learning the pythonic way of doing things.

@Zest

This moment right here is what I live for...

Give it time though you will start to have questions againa nd feel like it stops making sense. Dont let that win, you got the idea, it does make sense, and when you realize the "gotchas" it doesnt mean you dont really understand, it just means there are some unusual scenarios that require one other piece of the puzzle to understand and incorperate...

I dont want to break up your tears of joy but take this curve ball (again don't think it means your thinking about it wrong, i promise you aren't), i will give you the answer if you'd like but its kinda the only time piece you haven't considered so worth exploring...

Remember that ship looks short to you and the universe looks short to the ship... so take this class paradox of that scenario and consider how you might resolve it logically... You have a barn that normally is more than large enough to hold the ship. The barn has two huge doors on opposite sides. Normally the ship is small enough to fit inside. Now as a ship flying at 98% the speed of light flys past you and flys through the barn (the doors are initial open) you very quickly in the instant it is inside the barn close both the doors, containing it inside, then open them again before the ship crashes intot he door, and it flys away.

Now because the ship is going 98% the speed of light it is 19.9% its normal length. This isnt a problem because the ship could fit inside at full size anyway. However from the ships perspective the universe is 19.9% shorter and the ship is normal size. So that means from the perspective of the people on the ship the barn is 19.9% its normal length and they should not be able to fit inside it and have both doors closed at the same time. Yet somehow from their perspective it all works out.. how is this possible?

You can also flip this around and presume that at normal lengtht he ship is too big to fit in the barn but when it is traveling at speeds and is 19.9% its normal length it can fit inside just fine, therefore to the stationary person they can still close the doors safely when it is at speed, but couldnt otherwise, and to the ship there is still an apparent paradox (it isnt really but there is something you need to realize to understand why)...

Let me know if you have any ideas as to how this is answered, or if you just want me to answer it for you I Can do that too.

@Science

@freemo @Science oh my god! it all makes sense now I'm actually literally crying thank you so much for explaining all this!

@Zest

This moment right here is what I live for...

Give it time though you will start to have questions againa nd feel like it stops making sense. Dont let that win, you got the idea, it does make sense, and when you realize the "gotchas" it doesnt mean you dont really understand, it just means there are some unusual scenarios that require one other piece of the puzzle to understand and incorperate...

I dont want to break up your tears of joy but take this curve ball (again don't think it means your thinking about it wrong, i promise you aren't), i will give you the answer if you'd like but its kinda the only time piece you haven't considered so worth exploring...

Remember that ship looks short to you and the universe looks short to the ship... so take this class paradox of that scenario and consider how you might resolve it logically... You have a barn that normally is more than large enough to hold the ship. The barn has two huge doors on opposite sides. Normally the ship is small enough to fit inside. Now as a ship flying at 98% the speed of light flys past you and flys through the barn (the doors are initial open) you very quickly in the instant it is inside the barn close both the doors, containing it inside, then open them again before the ship crashes intot he door, and it flys away.

Now because the ship is going 98% the speed of light it is 19.9% its normal length. This isnt a problem because the ship could fit inside at full size anyway. However from the ships perspective the universe is 19.9% shorter and the ship is normal size. So that means from the perspective of the people on the ship the barn is 19.9% its normal length and they should not be able to fit inside it and have both doors closed at the same time. Yet somehow from their perspective it all works out.. how is this possible?

You can also flip this around and presume that at normal lengtht he ship is too big to fit in the barn but when it is traveling at speeds and is 19.9% its normal length it can fit inside just fine, therefore to the stationary person they can still close the doors safely when it is at speed, but couldnt otherwise, and to the ship there is still an apparent paradox (it isnt really but there is something you need to realize to understand why)...

Let me know if you have any ideas as to how this is answered, or if you just want me to answer it for you I Can do that too.

@Science

@nate_river I guess if you have no wish for people to develop cpu-limited apps on your language, then no. As you suggest and as @pganssle suggests you can (and perhaps should) write such things in C and just wrap them in python. That is a matter of opinion I suppose.

In my eyes the limitation here is not one that in any way improves python, its not a thing where they made python better in some ways by having the GIL lock issue and took that negative impact as a accepted consequence. So in that regard I do not find it excusable. It could have and should have been fixed with python3 and if they had the language would have been better (easier to write things that use multi cpus) without having any consequences... So to me and in my opinion, yes this is a black mark on the language and if they had it it would not have detracted from the intended purpose of python or made it any less suited at what its good at, it just would have made it more suited to a greater range of problems.

@Zest

People without science backgrounds is **exactly** who my OP is trying to target. It is people like you who have a curiosity about it, dont want to necessarily do the complex math, but do want to understand it in basic terms, that I feel are neglected.

It is a surprisingly simple idea to understand, but it is very rare that anyone who knows the topic will enough will actually both to explain it to someone like you in simple terms. So there is a gap of ignorance despite the curiosity and I dont like that, I dont like ignorance particularly when people are more than willing to try to understand.

So by all means **please** keep asking questions and take this conversation as far as you would like to get some understanding of the topic.

> I have one other question since you said something about being absolutely stationary and I was under the impression that this wasn't really a thing that exists

When i say absolutely stationary I mean that the distance and direction between you and some object are fixed. Better wording would have been "perfectly stationary", as in relative to someone or something else.

> I was under the impression that this wasn't really a thing that exists. are there actually stationary objects?

Your impression was correct there is no concept of stationary outside of relative terms.

> s there any such thing as absolute motion that's meaningful in a way that's not dependent upon it's relativity to other objects?

No there is not, not in any sense what so ever. It always must be relative to some thing. There is also no point that could be considered "the center of the universe" or even "where the big bang happened" which one might think if there were could be used as an absolute reference point. But no such point exists. The whole of the universe is where the big bang occurred more or less. The center of the universe, in any meaningful sense, is always wherever you happen to be, quite literally.

@Science

@pganssle

> In any case, I am not a Python maximalist. I suspect that you are not really giving Python a fair shot,

If i werent giving python a fair shot I wouldnt be using it and I wouldnt be rewriting my code 5x over to make it more pythonic where I can.

I am more than happy to keep trying to use it until it becomes impossible and dont intend to jump ship anytime soon. Likewise i will keep complaining and hold my criticisims until one of those rewrites, which hopefully get me closer to idiomatic python, produces code elegance and performance that I can praise.

For the moment the journey does not have me impressed, time will tell if that opinion changes. But for now I clearly am "giving it a chance", I just can and will continue to complain until those chances I keep giving it prove my preconceptions wrong (and maybe they will).

@Zest

When we talk abotu special relativity we usually dont talk about acceleration. We assume everyone has some speed and direction and that those speeds and direction are not changing. That is in fact what makes the difference between special relativity and general relativity. Once you get into acceleration then we have moved on to discussing the more complex ideas behind general relativity (though that is also where things get cool).

If you pretend acceleration doesnt have any effects and only think in terms of special relativity then under conditions where acceleration occurs the whole system breaks and you get logical paradoxes that dont make sense. General relativity resolves those paradoxes and describes the additional effects that take place when acceleration is introduced to the through experiment in some way.

@swiley @Science

@Zest @Science

> would those objects that are moving apart from each other anyways not still be moving apart from each other at rather high speed

from all observers the distance between the earth and original ship will appear to be increasing, though not at the same rate.

> if all three objects stopped moving relative to each other then, how would that effect the distance between them?

You have actually just hit on the third component of special relativity, you already knew about time dilation as you mentioned it, and we already discussed velocity dilation (that velocity is relative to the observer just like time is), now you are basically realizing the third aspect of special relativity, that is, length dilation (formally called length constriction). the length of a thing is just as relative as the rate of time or the speed of the thing.

Depends on who out of the group is doing the decelerating in order to create a relative velocity of 0 between them.. If you are standing in between two spaceships each going 98% the speed of light in opposite directions away from you, and they both decelerated equally until velocity was 0, then the distance between you and teh ships and between the two ships from your perspective would not change (however the length of the ships themselves would). When the ships are moving at high speed they would be much short in the direction they were moving, specifically 19.9% of their normal length, as if they were "compressed" in that one direction. As they come to a stop their length would expand to their normal length again.

However the perspective from the moving ship is much more interesting, assuming the ship is the one that did the accelerating the entire universe, along the direction that it is moving, would be 19.9% shorter than it appears to you, so a star that would normally had been 1 light year away is now only 0.199 light years away and from the perspective of the person on the ship would take 1/5th the time to get there that it would have taken light to do the journey.

Similarly if we had two ships moving away from you at 98% the speed of light the distance from the perspective of the ship is 19.9% less from the ship to you or the ship to the other ship than it appears to you. However the moment the ship accelerates and becomes stationary to you the universe will expand to its normal length again and the distances seen from the people on the ship would agree entirely with your own measurement of distance.

@Zest @Science

> would those objects that are moving apart from each other anyways not still be moving apart from each other at rather high speed

from all observers the distance between the earth and original ship will appear to be increasing, though not at the same rate.

> if all three objects stopped moving relative to each other then, how would that effect the distance between them?

You have actually just hit on the third component of special relativity, you already knew about time dilation as you mentioned it, and we already discussed velocity dilation (that velocity is relative to the observer just like time is), now you are basically realizing the third aspect of special relativity, that is, length dilation (formally called length constriction). the length of a thing is just as relative as the rate of time or the speed of the thing.

Depends on who out of the group is doing the decelerating in order to create a relative velocity of 0 between them.. If you are standing in between two spaceships each going 98% the speed of light in opposite directions away from you, and they both decelerated equally until velocity was 0, then the distance between you and teh ships and between the two ships from your perspective would not change (however the length of the ships themselves would). When the ships are moving at high speed they would be much short in the direction they were moving, specifically 19.9% of their normal length, as if they were "compressed" in that one direction. As they come to a stop their length would expand to their normal length again.

However the perspective from the moving ship is much more interesting, assuming the ship is the one that did the accelerating the entire universe, along the direction that it is moving, would be 19.9% shorter than it appears to you, so a star that would normally had been 1 light year away is now only 0.199 light years away and from the perspective of the person on the ship would take 1/5th the time to get there that it would have taken light to do the journey.

Similarly if we had two ships moving away from you at 98% the speed of light the distance from the perspective of the ship is 19.9% less from the ship to you or the ship to the other ship than it appears to you. However the moment the ship accelerates and becomes stationary to you the universe will expand to its normal length again and the distances seen from the people on the ship would agree entirely with your own measurement of distance.

@pganssle In this case I am already using numpy and scipy, and pandas which i think also has some low level bindings.

As far as I can tell none of them do parallel processing out of the box (if they did I probably wouldnt need to do it myself). There were some projects that add it on, parallellal or whatever it was called, modin, and I think one other. Tried all three none of them really helped and had marginal improvements.

With that said I **have** been able to multiprocess this and the performance has went from 2 hours to process a year simulation down to 10 minutes. Not as good as i could get with better parallelization tools, but good enough to work with it for now. My main issue is how convoluted multiprocessing is rather than if it can do it at all.

I tried Dask through modin and it didnt do much speedup, I may need to just bite the bullet and try pure Dask at this point, though im not sure if it will solve my problem. Even if i could get my numpy and pandas operations parallel the core of the issue is realtime processing of streaming data effectively and neither numpy nor pandas handles that well, which suggests to me Dask either wont fix the issue either or if it does it will be even more convoluted than just using multiprocessing.

Either way, for now i am able to get things working with multiprocessing but I really dont like what the code looks like to get it there, or the compromises I had to make that limit me from seeing better performance.

@pganssle Oh and as for dropping into a low level language, if i were to do that, particularly on this project where the multiprocessing component cant be easily isolated fromt eh rest, then there would be very little if any of what im doing actually in python.

At that point I would just pick a language that didnt have these issues but still high level and avoid python all together. If i wanted to code in C I would have.

@pganssle Yea I suppose I should have kept my criticism to the language and the tech and not extend it to the developers, just in the sense of disrespect. So I will take that aspect.

But I stand by what I have said, I have worked with countless languages and know hundreds and of all the relatively mainstream and significantly developed languages out there I've never found a language that feels as sloppily put together as python does.

Don't get me wrong I've used python lightly for years. It has a place and for some things it works well enough to be a non-issue. But overall it feels like it wasnt developed to the level of quality i expect of a mainstream language.

@nate_river I dont think they have any intent. They missed their chance in the move from python2 to python3 which broke backwards compatibility and hopefully they have no intention of doing that a second time, so kinda damned if they do damned if they dont at this point.

With that said it was very doable they just choose not to do it. Even before they wrote python3 there were unofficial python interpreters that already fixed this problem and were drop-in compatible with the official python. the truth is they were just too lazy to do it.

Have I mentioned yet today how much I hate python multiprocessing, or writing cpu-limited algorithms on python in general... my god why could they not fix the GIL lock problem in python3, are the devs really this lazy!

@swiley

After you try to work through it and can formulate your confusion (assuming its still there) feel free to make a public post to the group and myself and others can do our best to try to explain it hopefully... though i wouldnt really hope for too much of an intuitive understanding of tensors honestly, you sort get one a little if you work with it enough but I think thats more just repetition and seeing the patterns than it is actually understanding it :)

@PauITown @Zest

Show older
CleverLibre Social

CleverLibre Social is an inclusive social instance for open discussion, learning, and community.
All cultures welcome.
Hate speech and harassment strictly forbidden.